```python
import numpy as np
import sympy as smp
from sympy.vector import *
import matplotlib.pyplot as plt
from scipy.integrate import quad
from scipy.integrate import quad_vec
```

```python
x, y, z, u1, u2, u3, v1, v2, v3, t = smp.symbols('x y z u_1 u_2 u_3 v_1 v_2 v_3 t')
```

# Vectors and Geometry

Numerical or symbolic

```python
a = np.array([1,2,3])
b = np.array([7,8,9])
u = smp.Matrix([u1,u2,u3])
v = smp.Matrix([v1,v2,v3])
```

## Addition and multiplication

```python
2*a+b
```

    array([ 9, 12, 15])

```python
2*u+v
```

$\displaystyle \left[\begin{matrix}2 u_{1} + v_{1}\\2 u_{2} + v_{2}\\2 u_{3} + v_{3}\end{matrix}\right]$

## Dot products

```python
np.dot(a, b)
```

    50

```python
u.dot(v)
```

$\displaystyle u_{1} v_{1} + u_{2} v_{2} + u_{3} v_{3}$

## Cross products

```python
np.cross(a,b)
```

    array([-6, 12, -6])

```python
u.cross(v)
```

$\displaystyle \left[\begin{matrix}u_{2} v_{3} - u_{3} v_{2}\\- u_{1} v_{3} + u_{3} v_{1}\\u_{1} v_{2} - u_{2} v_{1}\end{matrix}\right]$

## Length of vector

```python
np.linalg.norm(a)
```

    3.7416573867739413

```python
u.norm()
```

$\displaystyle \sqrt{\left|{u_{1}}\right|^{2} + \left|{u_{2}}\right|^{2} + \left|{u_{3}}\right|^{2}}$

## Vector projection

$\text{proj}_v(u) = \frac{u \cdot v}{|v|^2} v$

```python
proj_b_a = np.dot(a,b)/np.linalg.norm(b)**2 * b
proj_b_a
```

    array([1.80412371, 2.06185567, 2.31958763])

```python
proj_v_u = u.dot(v)/v.norm()**2 * v
proj_v_u
```

$\displaystyle \left[\begin{matrix}\frac{v_{1} \left(u_{1} v_{1} + u_{2} v_{2} + u_{3} v_{3}\right)}{\left|{v_{1}}\right|^{2} + \left|{v_{2}}\right|^{2} + \left|{v_{3}}\right|^{2}}\\\frac{v_{2} \left(u_{1} v_{1} + u_{2} v_{2} + u_{3} v_{3}\right)}{\left|{v_{1}}\right|^{2} + \left|{v_{2}}\right|^{2} + \left|{v_{3}}\right|^{2}}\\\frac{v_{3} \left(u_{1} v_{1} + u_{2} v_{2} + u_{3} v_{3}\right)}{\left|{v_{1}}\right|^{2} + \left|{v_{2}}\right|^{2} + \left|{v_{3}}\right|^{2}}\end{matrix}\right]$

## Lines/Planes in Space

Lines: $\vec{r}(t) = \vec{r}_0 + t\vec{v}$

```python
r0 = smp.Matrix([1,1,1])
v = smp.Matrix([1,3,-1])
r = r0 + t*v
r
```

$\displaystyle \left[\begin{matrix}t + 1\\3 t + 1\\1 - t\end{matrix}\right]$

Planes: $\vec{n} \cdot (P_0-\left<x,y,z\right>) = 0$

```python
P0 = smp.Matrix([4,4,8])
r = smp.Matrix([x,y,z])
n = smp.Matrix([1,1,1])
n.dot(P0-r)
```

$\displaystyle - x - y - z + 16$

**Examples** Find vector parallel to the line of intersection of the two planes $3x-6y-2z = 15$ and $2x+y-2z=5$. (It's  going to be perpendicular to both normal vectors)

```python
n1 = np.array([-3,6,2])
n2 = np.array([-2,-1,2])
ans = np.cross(n1,n2)
ans
```

    array([14,  2, 15])

# Vector Calculus

## Vector Derivatives

```python
r = smp.Matrix([3*t, smp.sin(t), t**2])
r
```

$\displaystyle \left[\begin{matrix}3 t\\\sin{\left(t \right)}\\t^{2}\end{matrix}\right]$

```python
smp.diff(r,t)
```

$\displaystyle \left[\begin{matrix}3\\\cos{\left(t \right)}\\2 t\end{matrix}\right]$

Example: Find the angle between the velocity and acceleration as a function of time $\theta(t)$

```python
v = smp.diff(r,t)
a = smp.diff(v,t)
theta = smp.acos(v.dot(a)/(v.norm()*a.norm())).simplify()
```

```python
theta
```

$\displaystyle \operatorname{acos}{\left(\frac{4 t - \frac{\sin{\left(2 t \right)}}{2}}{\sqrt{\left|{\sin{\left(t \right)}}\right|^{2} + 4} \sqrt{4 \left|{t}\right|^{2} + \left|{\cos{\left(t \right)}}\right|^{2} + 9}} \right)}$

```python
theta.subs(t,6).evalf()
```

$\displaystyle 0.251108015692338$

```python
tt = np.linspace(0, 10, 100)
aa = smp.lambdify([t], theta)(tt)
plt.plot(tt,aa)
plt.xlabel('$t$', fontsize=20)
plt.ylabel(r'$\theta(t)$', fontsize=20)
plt.show()
```

## Vector Integrals

Note: Will not add the +C

```python
r = smp.Matrix([smp.exp(t)*smp.cos(t), t**4, 1/(1+t**2)])
r
```

$\displaystyle \left[\begin{matrix}e^{t} \cos{\left(t \right)}\\t^{4}\\\frac{1}{t^{2} + 1}\end{matrix}\right]$

```python
smp.Integral(r).doit()
```

$\displaystyle \left[\begin{matrix}\frac{e^{t} \sin{\left(t \right)}}{2} + \frac{e^{t} \cos{\left(t \right)}}{2}\\\frac{t^{5}}{5}\\\operatorname{atan}{\left(t \right)}\end{matrix}\right]$

In some cases integral cannot be evaluated symbolically: must solve definite integrals numerically:

```python
r = smp.Matrix([smp.exp(t**2)*smp.cos(t)**3, smp.exp(-t**4), 1/(3+t**2)])
r
```

$\displaystyle \left[\begin{matrix}e^{t^{2}} \cos^{3}{\left(t \right)}\\e^{- t^{4}}\\\frac{1}{t^{2} + 3}\end{matrix}\right]$

Integrate from $t=0$ to $t=4$

```python
r_num = smp.lambdify([t], r)
```

```python
r_num = smp.lambdify([t], r)
quad_vec(r_num, 0, 4)[0]
```

    array([[-4.83559254e+05],
           [ 9.06402477e-01],
           [ 6.70972506e-01]])

## Arclength

$$L = \int_a^b \sqrt{dx^2 + dy^2 + dz^2} = \int_a^b \sqrt{(dx/dt)^2 + (dy/dt)^2 + (dz/dt)^2 } dt $$

Rare Cases can be done symbolically. Find arclength of $\left< 0, t, t^2 \right>$ from $t=0$ to $t=1$

```python
r = smp.Matrix([0, t, t**2])
smp.integrate(smp.diff(r,t).norm(), (t,0,1))
```

$\displaystyle \frac{\operatorname{asinh}{\left(2 \right)}}{4} + \frac{\sqrt{5}}{2}$

In most cases needs to be done numerically. Find arclength of $\left< e^t, \sin(t), t^4 \right>$ from $t=0$ to $t=1$

```python
r = smp.Matrix([smp.exp(t), smp.sin(t), t**4])
r_num = smp.lambdify([t], smp.diff(r,t).norm())
quad(r_num, 0, 1)[0]
```

    2.316977146444862

## Other Relavent Quantities

If $ds = \sqrt{dx^2 + dy^2 + dz^2}$ is the arclength element $ds/dt = \sqrt{(dx/dt)^2 + (dy/dt)^2 + (dz/dt)^2 } = |d\vec{r}/dt| = |\vec{v}|$ then the other quantities of importance are

* Unit Tangent vector: $\vec{T} = \frac{d\vec{r}}{dt}\frac{1}{|d\vec{r}/dt|} = \vec{v}/|\vec{v}|$
* Curvature: $\kappa = |\frac{d\vec{T}}{dt}|\frac{1}{|\vec{v}|}$
* Unit normal vector to Curve: $\vec{N} = \frac{d\vec{T}/dt}{|d\vec{T}/dt|}$

**Example** Find all these for $\vec{r}(t) = \left<a\cos(t)e^t, b\sin(t), ct\right>$

```python
t, a, b, c = smp.symbols('t a b c', pos=True, real=True)
```

Define the path

```python
r = smp.Matrix([a*smp.cos(t)*smp.exp(t), b*smp.sin(t), c*t])
r
```

$\displaystyle \left[\begin{matrix}a e^{t} \cos{\left(t \right)}\\b \sin{\left(t \right)}\\c t\end{matrix}\right]$

Define the velocity vector $\vec{v}$ and speed $|\vec{v}|$

```python
v = smp.diff(r, t)
v_norm = v.norm()
```

Get $\vec{T}$, $\kappa$, and $\vec{N}$

```python
T = v/v_norm
kappa = smp.diff(T, t).norm()/v_norm
N = smp.diff(T,t)/smp.diff(T,t).norm()
```

Can evaluate these quantities or make plots

```python
kappa.subs([(t,2),(a,3),(b,4),(c,5)]).evalf()
```

$\displaystyle 0.00775458595406284$

```python
N.subs([(t,2),(a,3),(b,4),(c,5)]).evalf()
```

$\displaystyle \left[\begin{matrix}-0.152946073876551\\-0.204518432770271\\-0.966840063890063\end{matrix}\right]$

Can also make continuous plots

```python
kappa_num = smp.lambdify([t,a,b,c], kappa)
```

```python
tt = np.linspace(0, 2, 100)
kk = kappa_num(tt, 1, 2, 3)
plt.plot(tt, kk)
plt.xlabel('$t$', fontsize=20)
plt.ylabel('$\kappa(t)$', fontsize=20)
plt.show()
```

# Partial/Directional Derivatives

```python
x, y, z = smp.symbols('x y z')
```

## Basics

Partial Derivatives $\partial f / \partial x$ and $\partial f / \partial y$ of $f(x,y)=y^2\sin(x+y)$

```python
f = y**2 * smp.sin(x+y)
smp.diff(f,x)
```

$\displaystyle y^{2} \cos{\left(x + y \right)}$

```python
smp.diff(f,y)
```

$\displaystyle y^{2} \cos{\left(x + y \right)} + 2 y \sin{\left(x + y \right)}$

Or more complicated things like $\frac{\partial^3 f}{\partial xy^2}$

```python
smp.diff(f,y,y,x)
```

$\displaystyle - y^{2} \cos{\left(x + y \right)} - 4 y \sin{\left(x + y \right)} + 2 \cos{\left(x + y \right)}$

## The Chain Rule

```python
t = smp.symbols('t')
x, y, z, w, v =smp.symbols('x, y, z w v', cls=smp.Function)
```

* Suppose $x$, $y$, and $z$ are functions of $t$
* $w$ is a function of $x$, $y$, and $z$. i.e $w=w(x,y,z)$

Find $dw/dt$

```python
x = x(t)
y = y(t)
z = z(t)
w = w(x, y, z)
```

Can keep things in a more functional form:

```python
smp.diff(w,t)
```

$\displaystyle \frac{d}{d x{\left(t \right)}} w{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)} \frac{d}{d t} x{\left(t \right)} + \frac{d}{d y{\left(t \right)}} w{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)} \frac{d}{d t} y{\left(t \right)} + \frac{d}{d z{\left(t \right)}} w{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)} \frac{d}{d t} z{\left(t \right)}$

Or put in specific functions:

```python
w1 = x**2+smp.exp(y)*smp.sin(z)
smp.diff(w1,t).subs([(x, smp.sin(t)),
                    (y, smp.cos(t)),
                    (z, t**2)]).doit()
```

$\displaystyle 2 t e^{\cos{\left(t \right)}} \cos{\left(t^{2} \right)} - e^{\cos{\left(t \right)}} \sin{\left(t \right)} \sin{\left(t^{2} \right)} + 2 \sin{\left(t \right)} \cos{\left(t \right)}$

## Gradients

Gradients: $\nabla f$. Now we are dealing with particular coordinate systems, so we must be careful:

```python
C = CoordSys3D('')
```

```python
f = C.x*smp.sin(C.y)
gradient(f)
```

$\displaystyle (\sin{\left(\mathbf{{y}_{}} \right)})\mathbf{\hat{i}_{}} + (\mathbf{{x}_{}} \cos{\left(\mathbf{{y}_{}} \right)})\mathbf{\hat{j}_{}}$

```python
gradient(f).to_matrix(C)
```

$\displaystyle \left[\begin{matrix}\sin{\left(\mathbf{{y}_{}} \right)}\\\mathbf{{x}_{}} \cos{\left(\mathbf{{y}_{}} \right)}\\0\end{matrix}\right]$

```python
gradient(f).to_matrix(C).subs([(C.y, 1)])
```

$\displaystyle \left[\begin{matrix}\sin{\left(1 \right)}\\\mathbf{{x}_{}} \cos{\left(1 \right)}\\0\end{matrix}\right]$

## Directional Derivatives

Directional derivatives $D_u f = \nabla f \cdot u$

```python
u = 4*C.i - 3*C.j + 2*C.k
# Make into unit vector
u = u.normalize()
u
```

$\displaystyle (\frac{4 \sqrt{29}}{29})\mathbf{\hat{i}_{}} + (- \frac{3 \sqrt{29}}{29})\mathbf{\hat{j}_{}} + (\frac{2 \sqrt{29}}{29})\mathbf{\hat{k}_{}}$

```python
gradient(f).dot(u)
```

$\displaystyle - \frac{3 \sqrt{29} \mathbf{{x}_{}} \cos{\left(\mathbf{{y}_{}} \right)}}{29} + \frac{4 \sqrt{29} \sin{\left(\mathbf{{y}_{}} \right)}}{29}$

## Extreme Values and Saddle Points

Extreme values of $f(x,y)$ can occur at

* 1. Boundary points of the domain of $f$
* 2. Critical Points ($f_x=f_y=0$)

If

* (i) $f_{xx}<0$ and $f_{xx}f_{yy} - f_{xy}^2>0$ at (a,b) then **local maximum**
* (ii) $f_{xx}>0$ and $f_{xx}f_{yy} - f_{xy}^2>0$ at (a,b) then **local minimum**
* (iii) $f_{xx}f_{yy} - f_{xy}^2 < 0$ at (a,b) then **saddle point**
* (iv) $f_{xx}f_{yy} - f_{xy}^2 = 0$ at (a,b) then **inconclusive**

```python
x, y = smp.symbols('x y', real=True)
f = x**3 + 3*x*y + y**3
f
```

$\displaystyle x^{3} + 3 x y + y^{3}$

```python
smp.solve([smp.diff(f,x), smp.diff(f, y)])
```

    [{x: -1, y: -1}, {x: 0, y: 0}]

```python
fxx = smp.diff(f,x,x)
fyy = smp.diff(f,y,y)
fxy = smp.diff(f,x,y)
```

Testing the first point

```python
fxx.subs([(x,-1),(y,-1)])
```

$\displaystyle -6$

```python
test = fxx*fyy-fxy**2
test.subs([(x,-1),(y,-1)])
```

$\displaystyle 27$

Thus the first point is a local maximum!

## Lagrange Multipliers

Minimize $f(x,y,z)$ subject to the constraint $g(x,y,z)=0$

* Requires solving the two equations $\nabla f = \lambda \nabla g$ and $g(x,y,z)=0$

**Example**: A space probe has the shape of an ellipsoid $4x^2 + y^2 + 4z^2 = 16$ and after sitting in the sun for an hour. the temperature on its surface is given by $T(x,y,z)=8x^2+4yz-16z+600$. Find the hottest point on the surface

Note that $g=4x^2 + y^2 + 4z^2 - 16 = 0$ and $f=T = 8x^2+4yz-16z+600$ here

```python
C = CoordSys3D('')
```

```python
eq1 = gradient(f)-lam*gradient(g)
eq1 = eq1.to_matrix(C)
```

```python
lam = smp.symbols('\lambda')
g = 4*C.x**2 + C.y**2 + 4*C.z**2 - 16
f = 8*C.x**2 + 4*C.y*C.z-16*C.z+600
eq1 = gradient(f)-lam*gradient(g)
eq1 = eq1.to_matrix(C)
eq2 = g
sols = smp.solve([eq1, eq2])
sols
```

    [{.x: -4/3, .y: -4/3, .z: -4/3, \lambda: 2},
     {.x: 0, .y: -2, .z: -sqrt(3), \lambda: sqrt(3)},
     {.x: 0, .y: -2, .z: sqrt(3), \lambda: -sqrt(3)},
     {.x: 0, .y: 4, .z: 0, \lambda: 0},
     {.x: 4/3, .y: -4/3, .z: -4/3, \lambda: 2}]

```python
for sol in sols:
    print(f.subs(sol).evalf())
```

    642.666666666667
    641.569219381653
    558.430780618347
    600.000000000000
    642.666666666667

First and last solutions are the points where the probe is the hottest.

# Multiple Integrals

In rare cases these can be done symbolically:

$$\int_0^1 \int_0^{1-x^2} \int_3^{4-x^2-y^2} xdzdydx$$

```python
x, y, z = smp.symbols('x, y, z')
f = x
smp.integrate(f, (z, 3, 4-x**2-y**2), (y, 0, 1-x**2), (x, 0, 1))
```

$\displaystyle \frac{1}{8}$

But most of the time they need to be done numerically, such as in this example:

$$\int_0^1 \int_0^{1-x^2} \int_3^{4-x^2-y^2} xe^{-y}\cos(z)dzdydx$$

```python
# Wont Run!!
#x, y, z = smp.symbols('x, y, z')
#f = x*smp.exp(-y)*smp.cos(z)
#smp.integrate(f, (z, 3, 4-x**2-y**2), (y, 0, 1-x**2), (x, 0, 1))
```

Use scipy to evaluate this numerically

```python
from scipy.integrate import tplquad
f = lambda z, y, x: x*np.exp(-y)*np.cos(z)
tplquad(f, 0, 1,
        lambda x: 0, lambda x: 1-x**2,
        lambda x, y: 3, lambda x, y: 4-x**2-y**2)[0]
```

    -0.09109526451447894

# Integrals and Vector Fields

## Line Integrals (Scalar)

Given $\vec{r}(t) = \left< g(t), h(t), k(t) \right>$ the line integral of $f(x,y,z)$ along the curve is

$$\int_C f(x,y,z) ds = \int_a^b f(g(t), h(t), k(t))|d\vec{r}/dt|dt$$

```python
t = smp.symbols('t', real=True)
x, y, z, f = smp.symbols('x y z f', cls=smp.Function, real=True)
x = x(t)
y = y(t)
z = z(t)
r = smp.Matrix([x, y, z])
f = f(x,y,z)
```

```python
integrand = f*smp.diff(r,t).norm()
integrand
```

$\displaystyle \sqrt{\left|{\frac{d}{d t} x{\left(t \right)}}\right|^{2} + \left|{\frac{d}{d t} y{\left(t \right)}}\right|^{2} + \left|{\frac{d}{d t} z{\left(t \right)}}\right|^{2}} f{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)}$

Suppose
* $\vec{r}(t) = \left<\cos(t), \sin(t), t \right>$
* $f(x,y,z)=2xy+\sqrt{z}$

and we are going from $t=0$ to $t=2\pi$. In this case it can be solved symbolically:

```python
integrand1 = integrand.subs([(f, 2*x*y+smp.sqrt(z)),
                            (x, smp.cos(t)),
                            (y, smp.sin(t)),
                            (z, t)]).doit().simplify()
smp.integrate(integrand1, (t, 0, 2*smp.pi))
```

$\displaystyle \frac{\sqrt{2}}{2} + \sqrt{2} \left(- \frac{1}{2} + \frac{4 \sqrt{2} \pi^{\frac{3}{2}}}{3}\right)$

In general, can't be done symbolically. Consider now

* $\vec{r}(t) = \left<3\cos(t), 2\sin(t), e^{t/4} \right>$
* $f(x,y,z)=2xy+\sqrt{z}$

and we are going from $t=0$ to $t=2\pi$.

```python
integrand2 = integrand.subs([(f, 2*x*y+smp.sqrt(z)),
                            (x, 3*smp.cos(t)),
                            (y, smp.sin(t)),
                            (z, smp.exp(t/4))]).doit().simplify()
# Won't Run!
#smp.integrate(integrand2, (t, 0, 2*smp.pi))
```

```python
integrand2
```

$\displaystyle \frac{\left(e^{\frac{t}{8}} + 3 \sin{\left(2 t \right)}\right) \sqrt{e^{\frac{t}{2}} + 128 \sin^{2}{\left(t \right)} + 16}}{4}$

Do it numerically using scipy's `quad` function

```python
integrand2_num = smp.lambdify([t], integrand2)
quad(integrand2_num, 0, 2*np.pi)[0]
```

    21.05530045675979

## Line Integrals (Vector)

Given $\vec{r}(t) = \left< g(t), h(t), k(t) \right>$ the line integral of $\vec{F}(x,y,z)$ along the curve is

$$\int_C \vec{F}(x,y,z) \cdot d\vec{r} = \int_a^b\vec{F}(g(t), h(t), k(t)) \cdot \frac{d\vec{r}}{dt} dt$$

```python
t = smp.symbols('t', real=True)
x, y, z, F1, F2, F3 = smp.symbols('x y z F_1, F_2, F_3', cls=smp.Function, real=True)
x = x(t)
y = y(t)
z = z(t)
F1 = F1(x,y,z)
F2 = F2(x,y,z)
F3 = F3(x,y,z)
r = smp.Matrix([x, y, z])
F = smp.Matrix([F1, F2, F3])
```

```python
integrand = F.dot(smp.diff(r,t))
integrand
```

$\displaystyle \operatorname{F_{1}}{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)} \frac{d}{d t} x{\left(t \right)} + \operatorname{F_{2}}{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)} \frac{d}{d t} y{\left(t \right)} + \operatorname{F_{3}}{\left(x{\left(t \right)},y{\left(t \right)},z{\left(t \right)} \right)} \frac{d}{d t} z{\left(t \right)}$

Suppose $\vec{F} = \left< \sqrt{z}, -2x, \sqrt{y}\right>$ and $\vec{r}(t) = \left<t, t^2, t^4 \right>$ from $t=0$ to $t=1$

```python
integrand1 = integrand.subs([(F1, smp.sqrt(z)),
                             (F2, -2*x),
                             (F3, smp.sqrt(y)),
                             (x, t),
                             (y, t**2),
                             (z, t**4)]).doit().simplify()
smp.integrate(integrand1, (t, 0, 1))
```

$\displaystyle - \frac{1}{5}$

Often times this can't be done by hand, and one needs to evaluate numerically using scipy:

Suppose $\vec{F} = \left< \sqrt{|z|}, -2x, \sqrt{|y|}\right>$ and $\vec{r}(t) = \left<3\cos^2(t), t^2, 2\sin(t) \right>$ from $t=0$ to $t=2\pi$

```python
integrand2 = integrand.subs([(F1, smp.sqrt(smp.Abs(z))),
                             (F2, -2*x),
                             (F3, smp.sqrt(smp.Abs(y))),
                             (x, 3*smp.cos(t)**2),
                             (y, t**2),
                             (z, 2*smp.sin(t))]).doit().simplify()
# Won't Run
#smp.integrate(integrand2, (t, 0, 2*smp.pi))
```

```python
integrand2
```

$\displaystyle 2 \left(- 6 t \cos{\left(t \right)} - 3 \sqrt{2} \sin{\left(t \right)} \sqrt{\left|{\sin{\left(t \right)}}\right|} + \left|{t}\right|\right) \cos{\left(t \right)}$

Do it numerically using scipy's `quad` function

```python
integrand2_num = smp.lambdify([t], integrand2)
quad(integrand2_num, 0, 2*np.pi)[0]
```

    -118.4352528130723

## Surface Integrals (Scalar)

Area of a surface parameterized by $\vec{r}(u,v)$ is given by

$$A = \iint_S \left| \frac{d\vec{r}}{du} \times \frac{d\vec{r}}{dv} \right| du dv$$

The surface integral of a scalar function $G(\vec{r})$ is given by

$$\iint_S G(\vec{r}(u,v)) \left| \frac{d\vec{r}}{du} \times \frac{d\vec{r}}{dv} \right| du dv$$

2D parabola: $\vec{r}(x,y) = \left<x, y, x^2+y^2 \right>$ and thus $\vec{r}(\rho, \theta) = \left<\rho\cos\theta, \rho\sin\theta, \rho^2\right>$. Suppose the surface density is given by $G(x,y,z)=x^2+y^2$. Find surface integral for $0<\rho<1$ and $0<\theta<2\pi$

```python
rho, the = smp.symbols(r'\rho \theta',pos=True, real=True)
x, y, z, G = smp.symbols('x y z G', cls=smp.Function, real=True)
x = x(rho, the)
y = y(rho, the)
z = z(rho, the)
G = G(x,y,z)
r = smp.Matrix([x, y, z])
```

```python
integrand = G*smp.diff(r,rho).cross(smp.diff(r,the)).norm()
integrand
```

$\displaystyle \sqrt{\left|{\frac{\partial}{\partial \rho} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \theta} y{\left(\rho,\theta \right)} - \frac{\partial}{\partial \theta} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \rho} y{\left(\rho,\theta \right)}}\right|^{2} + \left|{\frac{\partial}{\partial \rho} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \theta} z{\left(\rho,\theta \right)} - \frac{\partial}{\partial \theta} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \rho} z{\left(\rho,\theta \right)}}\right|^{2} + \left|{\frac{\partial}{\partial \rho} y{\left(\rho,\theta \right)} \frac{\partial}{\partial \theta} z{\left(\rho,\theta \right)} - \frac{\partial}{\partial \theta} y{\left(\rho,\theta \right)} \frac{\partial}{\partial \rho} z{\left(\rho,\theta \right)}}\right|^{2}} G{\left(x{\left(\rho,\theta \right)},y{\left(\rho,\theta \right)},z{\left(\rho,\theta \right)} \right)}$

```python
integrand1 = integrand.subs([(G, x**2+y**2),
                             (x, rho*smp.cos(the)),
                             (y, rho*smp.sin(the)),
                             (z, rho**2)]).doit().simplify()
integrand1
```

$\displaystyle \rho^{2} \sqrt{4 \rho^{2} + 1} \left|{\rho}\right|$

```python
smp.integrate(integrand1, (the, 0, 2*smp.pi), (rho, 0, 1))
```

$\displaystyle \frac{\pi}{60} + \frac{5 \sqrt{5} \pi}{12}$

As before, if this integral can't be done symbolically, need to use scipy to convert to numerical function.

## Surface Integrals (Vector)

The surface integral of a vector function $\vec{G}(\vec{r})$ is given by

$$\iint_S \vec{G}(\vec{r}(u,v)) \cdot \left(\frac{d\vec{r}}{du} \times \frac{d\vec{r}}{dv}\right) du dv$$

This is also know as the flux of the vector field $\vec{G}$ through the surface

2D parabola: $\vec{r}(x,y) = \left<x, y, x^2+y^2 \right>$ and thus $\vec{r}(\rho, \theta) = \left<\rho\cos\theta, \rho\sin\theta, \rho^2\right>$. Suppose a vector field is given by $\vec{G}(x,y,z)=\left<y^2, z, 0 \right>$. Find the flux of $\vec{G}$ through half the surface for $0<\rho<1$ and $0<\theta<\pi$

```python
rho, the = smp.symbols(r'\rho \theta',pos=True, real=True)
x, y, z, G1, G2, G3 = smp.symbols('x y z G_1, G_2, G_3', cls=smp.Function, real=True)
x = x(rho, the)
y = y(rho, the)
z = z(rho, the)
G1 = G1(x,y,z)
G2 = G2(x,y,z)
G3 = G3(x,y,z)
r = smp.Matrix([x, y, z])
G = smp.Matrix([G1, G2, G3])
```

```python
integrand = G.dot(smp.diff(r,rho).cross(smp.diff(r,the)))
integrand
```

$\displaystyle \left(\frac{\partial}{\partial \rho} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \theta} y{\left(\rho,\theta \right)} - \frac{\partial}{\partial \theta} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \rho} y{\left(\rho,\theta \right)}\right) \operatorname{G_{3}}{\left(x{\left(\rho,\theta \right)},y{\left(\rho,\theta \right)},z{\left(\rho,\theta \right)} \right)} + \left(- \frac{\partial}{\partial \rho} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \theta} z{\left(\rho,\theta \right)} + \frac{\partial}{\partial \theta} x{\left(\rho,\theta \right)} \frac{\partial}{\partial \rho} z{\left(\rho,\theta \right)}\right) \operatorname{G_{2}}{\left(x{\left(\rho,\theta \right)},y{\left(\rho,\theta \right)},z{\left(\rho,\theta \right)} \right)} + \left(\frac{\partial}{\partial \rho} y{\left(\rho,\theta \right)} \frac{\partial}{\partial \theta} z{\left(\rho,\theta \right)} - \frac{\partial}{\partial \theta} y{\left(\rho,\theta \right)} \frac{\partial}{\partial \rho} z{\left(\rho,\theta \right)}\right) \operatorname{G_{1}}{\left(x{\left(\rho,\theta \right)},y{\left(\rho,\theta \right)},z{\left(\rho,\theta \right)} \right)}$

```python
integrand1 = integrand.subs([(G1, y**2),
                             (G2, z),
                             (G3, 0),
                             (x, rho*smp.cos(the)),
                             (y, rho*smp.sin(the)),
                             (z, rho**2)]).doit().simplify()
integrand1
```

$\displaystyle - 2 \rho^{4} \left(\frac{\sin{\left(2 \theta \right)}}{2} + 1\right) \sin{\left(\theta \right)}$

```python
smp.integrate(integrand1, (the, 0, smp.pi), (rho, 0, 1))
```

$\displaystyle - \frac{4}{5}$

Again, if this could not be done symbolically, one would need to convert the integrand into a numpy function and integrate using scipys `quad` function, like in previous examples.

## Explicit Sympy Functionality

```python
from sympy.vector import ParametricRegion
```

Find the mass of a cylinder with radius $a$ and height $h$ centered at origin with density $\rho(x,y)=x^2+y^2$

```python
a, h, r, the, z = smp.symbols(r'a h r \theta z', positive=True)
cylinder = ParametricRegion((r*smp.cos(the), r*smp.sin(the), z),
                    (the, 0, 2*smp.pi), (z, 0, h), (r, 0, a))
vector_integrate(C.x**2 + C.y**2, cylinder)
```

$\displaystyle \frac{\pi a^{4} h}{2}$

Not necessarily a fan of this functionailty because there is no clear way of converting it into an numerical function if it cannot be computed symbolically. For more see

https://docs.sympy.org/latest/modules/vector/vector_integration.html


```python
from sympy import *
init_printing()
from IPython.display import display
import matplotlib.pyplot as plt
import mpl_toolkits.mplot3d as a3
import matplotlib.animation as animation
%matplotlib notebook
# specify a second time as a workaround to get interactive plots working
%matplotlib notebook
#reload(gaussian_orbitals)
import gaussian_orbitals
import read_qmcpack
#reload(read_qmcpack)
import numpy as np
import scipy.optimize
from ipywidgets import interact
import ipywidgets
```

### Cusp Correction for Gaussian Orbitals

From "Scheme for adding electron-nucleus cusps to Gaussian orbitals" A. Ma, D. Towler, N. D. Drummond, R. J. Needs, Journal of Chemical Physics 122, 224322(2005) https://doi.org/10.1063/1.1940588

```python
phi = Symbol('phi')
phi_t = Symbol('phitilde')
eta = Symbol('eta')
psi = Symbol('psi')
psi_t = Symbol('psitilde')
phi, phi_t, eta, psi, psi_t
```

Each orbital can be divided into two parts - the s-type functions on the current center, and everything else.
* $\phi$ The s-type functions on the current center (original functions, no cusp correction)
* $\eta$ All non-s-type functions on the current center, and all functions from other centers (no need for cusp correction)
* $\psi$ Total uncorrected orbital ($ = \phi  + \eta$)
* $\tilde{\phi}$  Cusp-corrected s-type functions on the current center
* $\tilde{\psi}$ Total cusp-corrected orbital ($ = \tilde{\phi} + \eta$)

Inside some cutoff radius ($r_c$) the s-type part of the orbital is replaced with (Eqn 7 in the paper):

```python
C = Symbol('C')
p = Symbol('p')
p_sym = p
r = Symbol('r',real=True,nonnegative=True)
R = Symbol('R')
eq_phi1 = Eq(phi_t, C + sign(phi_t(0))*exp(p(r)))
eq_R = Eq(R(r), sign(phi_t(0))*exp(p(r)))
eq_phi2 = Eq(phi_t, C + R(r))
display(eq_phi1)
display(eq_R)
display(eq_phi2)

```

```python
alpha = IndexedBase('alpha',shape=(5,))
```

Where $p$ is a polynomial with the $\alpha$'s as coefficients

```python
p = alpha[0] + alpha[1]*r + alpha[2]*r**2 + alpha[3]*r**3 + alpha[4]*r**4
Eq(p_sym, p)
```

```python
rc = Symbol('r_c')
```

```python
R_def = exp(p)
R_def
```

### Solve for polynomial coefficients
Now to express the $\alpha$'s in terms of various contraints on the wavefunction (The value of the wavefunction and derivatives at the constraint points are the $X$'s)

```python
X1,X2,X3,X4,X5 = symbols('X_1 X_2 X_3 X_4 X_5')
# Constraints
# Value of phi tilde matches orbital at r_c
eq1 = Eq(p.subs(r,rc), X1)
eq1
```

```python
# derivative of phi tilde matches orbital at r_c
eq2 = Eq(diff(p,r).subs(r,rc), X2)
eq2
```

```python
# 2nd derivative of phi tilde matches orbital at r_c
eq3 = Eq((diff(p,r,2)+diff(p,r)**2).subs(r,rc),X3)
eq3
```

```python
# Cusp condition - derivative at zero
eq4 = Eq(diff(p,r).subs(r,0),X4)
eq4
```

```python
# Value of phi tilde at 0
eq5 = Eq(p.subs(r,0),X5)
eq5
```

Solve for the polynomial coefficients ($\alpha$'s) in terms of the wavefunction and derivative values ($X$'s).   These should match Eqn 14  in the paper.

```python
sln = solve([eq1, eq2, eq3, eq4, eq5],[alpha[0], alpha[1], alpha[2], alpha[3], alpha[4]])[0]
sln
```

```python
Eq(alpha[2],simplify(sln[2]))
```

```python
Eq(alpha[3],expand(sln[3]))
```

```python
Eq(alpha[4],expand(sln[4]))
```

```python
# Expand in terms of X's
p_X = p.subs({alpha[i]:sln[i] for i in range(5)})
display(p_X)
c_p_X = expand(p_X)
for sym in [X5, X4, X3, X2, X1]:
    c_p_X = collect(c_p_X, sym)
display(c_p_X)
```

 ### Effective local energy

 Fit this to an 'ideal local energy' to get the final parameter

```python
def del_spherical(e, r):
    """Compute Laplacian for expression e with respect to symbol r.
        Currently works only with radial dependence"""
    t1 = r*r*diff(e, r)
    t2 = diff(t1, r)/(r*r)
    return simplify(t2)
```

```python
# Effective one-electron local energy
p_sym = Symbol('p')
phi_tilde = exp(p_sym(r))
Zeff = Symbol('Z_eff')
El = -S.Half * del_spherical(phi_tilde, r)/phi_tilde - Zeff/r
#print R_def
#print del_spherical(R_def, r)
display(El)
El_sym = El.subs(p_sym(r), p).doit()
El_sym
```

```python
def eval_local_energy(gto, alpha_vals, r_val, Zeff_val):
    slist = {alpha[0]:alpha_vals[0], alpha[1]:alpha_vals[1], alpha[2]: alpha_vals[2], alpha[3]:alpha_vals[3],
           alpha[4]:alpha_vals[4], Zeff:Zeff_val, r:r_val}
    return El_sym.subs(slist).evalf()
```

```python
def get_current_local_energy(gto, xs, rc_val, alpha_vals, Zeff_val):
    EL_curr = []
    EL_at_rc = eval_local_energy(gto, alpha_vals, rc_val, Zeff_val)
    dE = -EL_at_rc
    #print 'dE = ',dE
    for x in xs:
        if x < rc_val:
            el = eval_local_energy(gto, alpha_vals, x, Zeff_val)
            EL_curr.append(el + dE)
        else:
            val, grad, lap = [g[0] for g in gto.eval_vgl(x, 0.0, 0.0)]
            real_el = -.5*lap / val - Zeff_val/x
            EL_curr.append(real_el + dE)
    return EL_curr

```

### Evaluate for He orbital

```python
basis_set, he_MO = read_qmcpack.parse_qmc_wf('he_sto3g.wfj.xml',['He'])
he_gto = gaussian_orbitals.GTO(basis_set['He'])
rc_val = 0.1
he_Z_val = 2.0

```

    MO coeff size =  1

```python
xvals = np.linspace(start=-2.0, stop=2.0, num=40)
yvals = np.array([he_gto.eval_v(x, 0.0, 0.0)[0] for x in xvals])
he_gto.eval_v(1.1, 0.0, 0.0)
```

```python
plt.plot(xvals, yvals)
```

```python
def compute_EL(X5_val):
    xslist = {X1:X1_val, X2:X2_val, X3:X3_val, X4:X4_val, X5:X5_val, rc:rc_val}
    alpha_vals = [s.subs(xslist) for s in sln]
    aslist = {alpha0:alpha_vals[0], alpha1:alpha_vals[1], alpha2:alpha_vals[2], alpha3:alpha_vals[3],
          alpha4:alpha_vals[4], Zeff:Zeff_val}
    Elof_r = El.subs(aslist)
    return Elof_r
```

```python
xs = np.linspace(start=0.012, stop=1.2*rc_val, num=10)
xs
```

    array([0.012, 0.024, 0.036, 0.048, 0.06 , 0.072, 0.084, 0.096, 0.108,
           0.12 ])

Coefficients from the paper to fit an 'ideal' effective one-electron local energy

```python
beta0 = Symbol('beta_0')
beta_vals = [beta0, 3.25819, -15.0126, 33.7308, -42.8705, 31.2276, -12.1316, 1.94692]
```

```python
El_terms = [beta_vals[n]*r**(n+1) for n in range(1,8)]
EL_ideal_sym = beta0 + sum(El_terms)
EL_ideal_sym

```

```python
# Compute ideal local energy at a point
def compute_ideal_EL(r_val, Z_val, beta0_val=0.0):
    Z = Symbol('Z')
    slist = {beta0: beta0_val, Z:Z_val, r:r_val}
    return (Z*Z*EL_ideal_sym).subs(slist).evalf()
```

```python
# Choose beta_0
El_orig_at_rc = compute_ideal_EL(rc_val, he_Z_val)
Z_val = he_Z_val
print 'EL orig at r_c',El_orig_at_rc
beta0_val = -(El_orig_at_rc)/Z_val/Z_val
beta0_val
```

```python
EL_ideal = [compute_ideal_EL(rval,he_Z_val, beta0_val) for rval in xs]
EL_ideal
```

```python
# Evaluate values of X's
def evalX(phi_func, rc_val, C_val, Z_val, phi_at_zero, eta_at_zero=0.0):
    X = [0.0]*5
    phi_at_rc, grad_at_rc, lapl_at_rc = phi_func(rc_val)
    X[0] = log(abs(phi_at_rc - C_val))
    X[1] = grad_at_rc[0] / (phi_at_rc - C_val)
    X[2] = (lapl_at_rc - 2.0*grad_at_rc[0]/rc_val)/(phi_at_rc - C_val)
    X[3] = -Z_val * (phi_at_zero + eta_at_zero) / (phi_at_zero - C_val)
    X[4] = log(abs(phi_at_zero - C_val))
    return X
```

```python
def create_phi_func(gto):
    def phi_func(r_val):
        val,grad,lap = gto.eval_vgl(r_val, 0.0, 0.0)
        return val[0], grad[0], lap[0]

    return phi_func
```

```python
Xvals = [0.0]*5
C_val = 0.0
he_Z_val = 2.0
he_phi = create_phi_func(he_gto)
evalX(he_phi, rc_val, C_val, he_Z_val, he_phi(0.0)[0])
```

```python
def solve_for_alpha(Xvals):
    xslist = {X1:Xvals[0], X2:Xvals[1], X3:Xvals[2], X4:Xvals[3], X5:Xvals[4], rc:rc_val}
    alpha_vals = [s.subs(xslist) for s in sln]
    return alpha_vals
```

```python
he_alpha_vals = solve_for_alpha(Xvals)
```

```python
print rc_val
```

    0.1

```python
EL_curr = get_current_local_energy(he_gto, xs, rc_val, he_alpha_vals, he_Z_val)
EL_curr
```

```python
plt.plot(xs, EL_ideal, xs, EL_curr)
```

```python
def compute_chi2(EL_ideal, EL_curr):
    return sum([(e1-e2)**2 for e1,e2 in zip(EL_ideal, EL_curr)])
```

```python
compute_chi2(EL_ideal, EL_curr)
```

```python
def compute_one_cycle(phi_func, gto, rc_val, Z_val, phi_at_zero, eta_at_zero=0.0):
    C_val = 0.0
    X = evalX(phi_func, rc_val, C_val, Z_val, phi_at_zero, eta_at_zero)
    alpha_vals = solve_for_alpha(X)
    EL_curr = get_current_local_energy(he_gto, xs, rc_val, alpha_vals, Z_val)
    chi2 = compute_chi2(EL_ideal, EL_curr)
    return chi2, alpha_vals, EL_curr

```

```python
phi_at_zero = he_phi(0.0)[0]
EL_curr = []
for ioffset in range(10):
    chi2, alpha_vals, EL_curr = compute_one_cycle(he_phi, he_gto, rc_val, he_Z_val, phi_at_zero+.01*ioffset)
    print chi2
```

    25854.2846426019
    21111.8921728781
    16918.8592218111
    13256.3864130293
    10106.4199357371
    7451.61627411234
    5275.30889319427
    3561.47675650037
    2294.71455959869
    1460.20457213250

```python
# See the local energy and ideal local energy change as phi(0) changes

fig, ax = plt.subplots(1,2)
plt.subplots_adjust(wspace = 0.5)
chi2, alpha_vals, EL_curr = compute_one_cycle(he_phi, he_gto, rc_val, he_Z_val, phi_at_zero)
chi2 = float(chi2)
ax[0].plot(xs, EL_ideal,label="Ideal local energy")
ax[0].set_ylabel("Energy")
ax[0].set_xlabel("r")
line, = ax[0].plot(xs, EL_curr, label="Local energy")
ax[0].legend()
chis = [chi2]
offsets = [0.0]
ax[1].set_xlim(-0.01, 20*0.01)
ax[1].set_ylim(0.0, chi2)
ax[1].set_ylabel("$\chi^2$")
ax[1].set_xlabel("$\phi(0)$")
line_chi, = ax[1].plot(offsets, chis, 'bo')

def animate_chi2(ioffset):
    offset = ioffset*0.01
    chi2, alpha_vals, EL_curr = compute_one_cycle(he_phi, he_gto, rc_val, he_Z_val, phi_at_zero + offset)
    print chi2, offset
    line.set_ydata(EL_curr)
    offsets.append(offset)
    chis.append(chi2)
    line_chi.set_xdata(offsets)
    line_chi.set_ydata(chis)
    #line_chi.plot(offsets, chis)
    return line,

# Uncomment the following to see the animation
#ani = animation.FuncAnimation(fig, animate_chi2, np.arange(1,20), interval=100, blit=True, repeat=False)
#plt.show()
```

    <IPython.core.display.Javascript object>

```python
# Interactive plot with r_c and phi(0) adjustable
phi_slider = ipywidgets.FloatSlider(value=phi_at_zero, min=phi_at_zero/2.0, max=phi_at_zero*2.0)
rc_slider = ipywidgets.FloatSlider(value=rc_val,min=rc_val/1.5,max=rc_val*1.5)
print rc_val
#plt.plot(xs, EL_curr, xs, EL_ideal)
fig2 = plt.figure()
ax2 = fig2.add_subplot(1,1,1)
ax2.set_xlabel("r")
ax2.set_ylabel("Local energy")
line2, = ax2.plot(xs,EL_ideal)
line3, = ax2.plot(xs, EL_curr)

def update(phi0=1.0, rc_new=0.1):
    chi2, alpha_vals, EL_curr = compute_one_cycle(he_phi, he_gto, rc_new, he_Z_val, phi0)
    line3.set_ydata(EL_curr)
    fig2.canvas.draw()

# Uncomment to activate the interactive version
#interact(update, phi0 = phi_slider, rc_new=rc_slider)

```

```python
def chi2_opt(x):
    phi_at_zero = x[0]
    rc_val = x[1]
    chi2, alpha_vals, EL_curr = compute_one_cycle(he_phi, he_gto, rc_val, he_Z_val, phi_at_zero)
    return float(chi2)

phi_at_zero = float(he_phi(0.0)[0])
print 'starting phi(0) = ',phi_at_zero
# Optimize phi_0 and rc simultaneously
# This optimization to find the minimum chi2 can take a while.
scipy.optimize.minimize(chi2_opt,[phi_at_zero, rc_val])
```

    starting phi(0) =  0.999603733514

          fun: 7.740763089368863
     hess_inv: array([[ 4.37122586e-05, -3.53898845e-05],
           [-3.53898845e-05,  2.86520112e-05]])
          jac: array([-15512.01509583, -19156.76275462])
      message: 'Desired error not necessarily achieved due to precision loss.'
         nfev: 1128
          nit: 127
         njev: 279
       status: 2
      success: False
            x: array([0.66675309, 0.48046938])

### Dividing the wavefunction
 Into $\phi$ and $\eta$ pieces.  This is done in QMCPACK by writing zeros to the coefficient matrix.

```python
# For Neon with DEF2-SVP
ne_basis_set, ne_MO_matrix = read_qmcpack.parse_qmc_wf('ne_def2_svp.wfnoj.xml',['Ne'])
#for cg in ne_basis_set:
#    print cg

print ne_MO_matrix.shape
#ne_MO_matrix
ne_basis_set['Ne']
```

```python
c_phi = ne_MO_matrix.copy()
c_eta = ne_MO_matrix.copy()
basis_by_index = gaussian_orbitals.get_ijk_inverse_index(ne_basis_set['Ne'])
# Loop over MO
for mo_idx in range(ne_MO_matrix.shape[0]):

    for ao_idx in range(ne_MO_matrix.shape[1]):
        # Loop over centers (for Ne atom, there is only one)
        # If s-type
        basis_set, angular_info = basis_by_index[mo_idx]
        if basis_set.orbtype == 0:
            # s-type, part of phi but not eta
            c_eta[mo_idx, ao_idx] = 0.0
        else:
            # not s-type, part of eta but not phi
            c_phi[mo_idx, ao_idx] = 0.0
```

```python
c_phi
```

    array([[ 0.990133, -0.031233,  0.009072,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.322026,  0.643777,  0.460567, -0.      , -0.      , -0.      ,
            -0.      , -0.      , -0.      , -0.      , -0.      , -0.      ,
            -0.      , -0.      , -0.      ],
           [-0.      , -0.      , -0.      ,  0.697249, -0.      , -0.      ,
            -0.454527, -0.      , -0.      , -0.      , -0.      , -0.      ,
            -0.      , -0.      , -0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ],
           [ 0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ,  0.      ,  0.      ,  0.      ,
             0.      ,  0.      ,  0.      ]])

```python
ne_phi_mo = gaussian_orbitals.MO(gaussian_orbitals.GTO(ne_basis_set['Ne']), c_phi)
ne_phi_mo.eval_v(0.0, 0.0, 0.0)
```

    array([-16.11819129,  -4.48709323,   0.        ,   0.        ,
             0.        ,   0.        ,   0.        ,   0.        ,
             0.        ,   0.        ,   0.        ,   0.        ,
             0.        ,   0.        ])

```python
ne_eta_mo = gaussian_orbitals.MO(gaussian_orbitals.GTO(ne_basis_set['Ne']), c_eta)
ne_eta_mo.eval_v(0.0, 0.0, 0.0)
```

    array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
           -4.47043866,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ])

```python
mo_idx = 0
gto1 = gaussian_orbitals.GTO(ne_basis_set['Ne'][mo_idx:mo_idx+1])
ne_phi_mo1 = gaussian_orbitals.MO(gaussian_orbitals.GTO(ne_basis_set['Ne']), c_phi[mo_idx:mo_idx+1,:])
print ne_phi_mo1.eval_v(0.0, 0.0, 0.0)

ne_eta_mo1 = gaussian_orbitals.MO(gaussian_orbitals.GTO(ne_basis_set['Ne']), c_eta[mo_idx:mo_idx+1,:])
print ne_eta_mo1.eval_v(0.0, 0.0, 0.0)
```

    [-16.11819129]
    [0.]

```python
Xvals = [0.0]*5
C_val = 0.0
ne_Z_val = 10.0
ne_phi = create_phi_func(ne_phi_mo1)
ne_eta_at_zero = ne_eta_mo1.eval_v(0.0,0.0,0.0)[0]
evalX(ne_phi, rc_val, C_val, ne_Z_val, ne_phi(0.0)[0], ne_eta_at_zero)
```

```python

```

```python

```


```python
%matplotlib inline
```

# Swirl

Image swirling is a non-linear image deformation that creates a whirlpool
effect.  This example describes the implementation of this transform in
``skimage``, as well as the underlying warp mechanism.

## Image warping
When applying a geometric transformation on an image, we typically make use of
a reverse mapping, i.e., for each pixel in the output image, we compute its
corresponding position in the input.  The reason is that, if we were to do it
the other way around (map each input pixel to its new output position), some
pixels in the output may be left empty.  On the other hand, each output
coordinate has exactly one corresponding location in (or outside) the input
image, and even if that position is non-integer, we may use interpolation to
compute the corresponding image value.

## Performing a reverse mapping
To perform a geometric warp in ``skimage``, you simply need to provide the
reverse mapping to the :py:func:`skimage.transform.warp` function. E.g., consider
the case where we would like to shift an image 50 pixels to the left. The reverse
mapping for such a shift would be::

    def shift_left(xy):
        xy[:, 0] += 50
        return xy

The corresponding call to warp is::

    from skimage.transform import warp
    warp(image, shift_left)

## The swirl transformation
Consider the coordinate $(x, y)$ in the output image.  The reverse
mapping for the swirl transformation first computes, relative to a center
$(x_0, y_0)$, its polar coordinates,

\begin{align}\\theta = \\arctan(y/x)

    \\rho = \sqrt{(x - x_0)^2 + (y - y_0)^2},\end{align}

and then transforms them according to

\begin{align}r = \ln(2) \, \mathtt{radius} / 5

    \phi = \mathtt{rotation}

    s = \mathtt{strength}

    \\theta' = \phi + s \, e^{-\\rho / r + \\theta}\end{align}

where ``strength`` is a parameter for the amount of swirl, ``radius`` indicates
the swirl extent in pixels, and ``rotation`` adds a rotation angle.  The
transformation of ``radius`` into $r$ is to ensure that the
transformation decays to $\\approx 1/1000^{\mathsf{th}}$ within the
specified radius.

```python
import matplotlib.pyplot as plt

from skimage import data
from skimage.transform import swirl

image = data.checkerboard()
swirled = swirl(image, rotation=0, strength=10, radius=120)

fig, (ax0, ax1) = plt.subplots(nrows=1, ncols=2, figsize=(8, 3),
                               sharex=True, sharey=True)

ax0.imshow(image, cmap=plt.cm.gray)
ax0.axis('off')
ax1.imshow(swirled, cmap=plt.cm.gray)
ax1.axis('off')

plt.show()
```


# Classification

The goal of classification is to find a function that separates the data into positive/negative labels. In the case of a linear classifier, this reduces to finding a set of parameters $w^\star$ such that, $$ \begin{align} w^\star &= \arg \min_w \sum_{i=1}^{N} \left[y_i\neq \text{sign} (w^\top x_i) \right] \\ &= \arg \min_w \sum_{i=1}^{N} l_{0/1} (w; x_i, y_i) \end{align}.$$

The problem with the $l_{0/1}$ loss, is that it is non-convex (and non-differentiable), hence other surrogate losses must be used to optimize the number of points.

```python
% matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from util import gradient_descent, generate_linear_separable_data, generate_circular_separable_data
import plot_helpers
import IPython

% load_ext autoreload
% autoreload 2
import ipywidgets
from ipywidgets import interact, interactive, interact_manual

import pylab
pylab.rcParams['figure.figsize'] = (10, 5)

from classifiers import Perceptron, SVM, Logistic
from regularizers import L1Regularizer, L2Regularizer

```

```python
num_points = 100  # Number of points per class
noise = 0.5  # Noise Level (needed for data generation).

X, Y = generate_linear_separable_data(num_points, noise=noise, dim=2)

fig = plt.subplot(111)
opt = {'marker': 'ro', 'label': '+', 'size': 8}
plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8, 'legend': True}
plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

```

```python
# Separate into train and test sets!
np.random.seed(42)
indexes = np.arange(0, 2*num_points, 1)
np.random.shuffle(indexes)
num_train = int(np.ceil(2*.05*num_points))

X_train = X[indexes[:num_train]]
Y_train = Y[indexes[:num_train]]

X_test = X[indexes[num_train:]]
Y_test = Y[indexes[num_train:]]

fig = plt.subplot(111)

opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=fig, options=opt)

opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8,
       'x_label': '$x$', 'y_label': '$y$', 'legend': True}
plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=fig, options=opt)

```

## The Perceptron Algorithm

The perceptron loss is defined as: $$L(w; X, Y) =  \sum_{i=1}^{N} L_p(w; x_i, y_i) = \sum_{i=1}^{N} \max \{ 0, -y_i w^\top x_i \}.$$

The loss function is continuous, but not differentialbe at $y_i w^\top x_i=0$. The subgradient, however, exists and hence (stochastic) gradient descent converges. The subgradient is:

$$ \partial L_p(w; x_i,y_i) = \left\{\begin{array}{cc} 0 & \text{if } -y_i w^\top x_i < 0 \\ -y_i x_i & \text{if } -y_i w^\top x_i > 0 \\ \left[0, -y_i x_i \right] & \text{if } -y_i w^\top x_i = 0 \end{array}  \right. $$

```python
def change_learning_params(reg, eta0, n_iter, batch_size):
    classifier = Perceptron(X_train, Y_train)
    classifier.load_test_data(X_test, Y_test)

    regularizer = L2Regularizer(np.power(10., reg))
    np.random.seed(42)
    w0 = np.random.randn(3, )

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X_train.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': None,
            }
    trajectory, indexes = gradient_descent(w0, classifier, regularizer, opts)

    contour_plot = plt.subplot(121)
    error_plot = plt.subplot(122)

    opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=contour_plot, options=opt)

    opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=contour_plot, options=opt)

    contour_opts = {'n_points': 50, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': True, 'n_classes': 4}
    error_opts = {'epoch': 5, 'x_label': '$t$', 'y_label': 'error'}

    opts = {'contour_opts': contour_opts, 'error_opts': error_opts}
    plot_helpers.classification_progression(X, Y, trajectory, indexes, classifier,
                                            contour_plot=contour_plot, error_plot=error_plot,
                                            options=opts)

interact_manual(change_learning_params,
                reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
                eta0=ipywidgets.FloatSlider(value=1,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=100,
                                            min=5,
                                            max=100,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=1,
                                            min=1,
                                            max=X_train.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False)
               );
```

## The SVM Algorithm

The svm loss is defined as: $$L(w; X, Y) =  \sum_{i=1}^{N} L_{\text{svm}} (w; x_i, y_i) =  \sum_{i=1}^{N} \max \{ 0, 1-y_i w^\top x_i \}.$$

The loss function is continuous, but not differentialbe at $y_i w^\top x_i=0$. The subgradient, however, exists and hence (stochastic) gradient descent converges. The subgradient is:

$$ \partial L_{\text{svm}}(w;x_i,y_i) = \left\{\begin{array}{cc} 0 & \text{if } 1-y_i w^\top x_i < 0 \\ -y_i x_i & \text{if } 1-y_i w^\top x_i > 0 \\ \left[0, -y_i x_i \right] & \text{if } 1-y_i w^\top x_i = 0 \end{array}  \right. $$

The difference with the perceptron loss is that the SVM loss includes a loss margin.

```python
def change_learning_params(reg, eta0, n_iter, batch_size):
    classifier = SVM(X_train, Y_train)
    classifier.load_test_data(X_test, Y_test)

    regularizer = L2Regularizer(np.power(10., reg))
    np.random.seed(42)
    w0 = np.random.randn(3, )

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X_train.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': None
            }

    trajectory, indexes = gradient_descent(w0, classifier, regularizer, opts)

    contour_plot = plt.subplot(121)
    error_plot = plt.subplot(122)

    opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=contour_plot, options=opt)

    opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=contour_plot, options=opt)

    contour_opts = {'n_points': 100, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': True, 'n_classes': 4}
    error_opts = {'epoch': 5, 'x_label': '$t$', 'y_label': 'error'}

    opts = {'contour_opts': contour_opts, 'error_opts': error_opts}
    plot_helpers.classification_progression(X, Y, trajectory, indexes, classifier,
                                            contour_plot=contour_plot, error_plot=error_plot,
                                            options=opts)

interact_manual(change_learning_params,
                reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
                eta0=ipywidgets.FloatSlider(value=1,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=50,
                                            min=5,
                                            max=100,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=1,
                                            min=1,
                                            max=X_train.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False)
               );
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

## The Logistic Regression Algorithm

The logistic loss is defined as: $$L(w; X, Y) = \sum_{i=1}^{N} L_{\sigma}(w; x_i, y_i) = \sum_{i=1}^{N} \log \left( 1 + \exp \left(-y_i w^\top x_i \right) \right) .$$

The loss function is differentiable, hence the gradient is:

$$ \frac{\partial L_{\sigma}(w; x_i, y_i)}{\partial w} = -\frac{\exp \left(-y_i w^\top x_i \right)}{1+\exp \left(-y_i w^\top x_i \right)} y x_i$$

```python
def change_learning_params(reg, eta0, n_iter, batch_size):
    classifier = Logistic(X_train, Y_train)
    classifier.load_test_data(X_test, Y_test)

    regularizer = L2Regularizer(np.power(10., reg))
    np.random.seed(42)
    w0 = np.random.randn(3, )

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X_train.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': 'AdaGrad'
            }

    trajectory, indexes = gradient_descent(w0, classifier, regularizer, opts)

    contour_plot = plt.subplot(121)
    error_plot = plt.subplot(122)

    opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=contour_plot, options=opt)

    opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=contour_plot, options=opt)

    contour_opts = {'n_points': 100, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': True, 'n_classes': 4}
    error_opts = {'epoch': 5, 'x_label': '$t$', 'y_label': 'error'}

    opts = {'contour_opts': contour_opts, 'error_opts': error_opts}
    plot_helpers.classification_progression(X, Y, trajectory, indexes, classifier,
                                            contour_plot=contour_plot, error_plot=error_plot,
                                            options=opts)

interact_manual(change_learning_params,
                reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
                eta0=ipywidgets.FloatSlider(value=0.5,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=100,
                                            min=5,
                                            max=50,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=1,
                                            min=1,
                                            max=X_train.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False)
               );
```

# Kernelized Classification

Both perceptron and SVM can be kernelized, i.e. the output of the regressor can be passed through the sign function.

$$ \hat{y} = \text{sign} \left[ \sum_{i=1}^N \alpha_i y_i K(x_i, x) \right] $$ and a perceptron loss applied to the difference between the predictor and the true value.

The perceptron loss can be posed as:

$$ L(\alpha;x_j, y_j) = \max \left\{0, -\sum_{j=1}^N y_j \alpha_i K(x_i, x_j) \right\} $$

The parameters are updated as follows:
If $\hat{y}(x) \neq y(x)$, set $\alpha \gets \alpha + \eta_t$.

```python
num_points = 100  # Number of points per class
noise = 0.2  # Noise Level (needed for data generation).

X, Y = generate_circular_separable_data(num_points, noise=noise, offset=1)
fig = plt.subplot(111)
opt = {'marker': 'r*', 'label': '+'}
plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'label': '-'}
plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)
fig.legend();

```

```python
from kernels import PolynomialKernel, LinearKernel, LaplacianKernel, GaussianKernel, GaussianLinearKernel, PeriodicKernel, PeriodicLinearKernel

def change_learning_params(eta0, n_iter, batch_size, kernel, deg):
    if kernel == 'Linear':
        classifier = PolynomialKernel(X, Y, reg=0.00, deg=1, prediction=False)
    elif kernel == 'Polynomial':
        classifier = PolynomialKernel(X, Y, reg=0.00, deg=deg, prediction=False)
    elif kernel == 'Gaussian':
        classifier = GaussianKernel(X, Y, reg=0.00, bw=0.2, prediction=False)
    elif kernel == 'Laplacian':
        classifier = LaplacianKernel(X, Y, reg=0.00, bw=0.2, prediction=False)

    regularizer = L2Regularizer(0.)

    alpha0 = 0 * np.random.randn(X.shape[0])

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': None
            }
    alphas, indexes = gradient_descent(alpha0, classifier, regularizer, opts)

    fig = plt.subplot(111)
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-'}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

    contour_opts = {'n_points': 20, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': False, 'n_classes': 2}
    opts = {'contour_opts': contour_opts}
    plot_helpers.classification_progression(X, Y, alphas, indexes, classifier, contour_plot=fig, options=opts)

interact_manual(change_learning_params,
                eta0=ipywidgets.FloatSlider(value=0.5,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=20,
                                            min=5,
                                            max=50,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=X.shape[0],
                                            min=1,
                                            max=X.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                kernel=ipywidgets.RadioButtons(
                             options=['Polynomial', 'Gaussian', 'Laplacian'],
                             value='Polynomial',
                             description='Kernel type:',
                             style={'description_width': 'initial'}),
                deg = ipywidgets.IntSlider(
                         value=1,
                         min=1,
                         max=10,
                         step=1,
                         description='Degree of Polynomial kernel:',
                         style={'description_width': 'initial'}),
               );
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

## SVM KERNELS
The SVM can also be kernelized! let's look at an example

```python
def laplacian_kernel(X, Y, bw):
    rows = X.shape[0]
    cols = Y.shape[0]
    K = np.zeros((rows, cols))
    for col in range(cols):
        dist = bw * np.linalg.norm(X - Y[col, :], ord=1, axis=1)
        K[:, col] = np.exp(-dist)
    return K

```

```python
from sklearn import svm
from sklearn import datasets
import sklearn

# Our dataset and targets
n_samples = 200  # Number of points per class
tol = 1e-1
# noise = 0.2  # Noise Level (needed for data generation).

# X, Y = generate_circular_separable_data(num_points, noise=noise, offset=1)
# X = X[:, 0:2]

def svm_kernel(dataset, kernel, reg, bw, deg, noise):
    if dataset is 'blobs':
        X, Y = datasets.make_blobs(n_samples=n_samples, centers=2, random_state=3, cluster_std=10*noise)
    elif dataset is 'circles':
        X, Y = datasets.make_circles(n_samples=n_samples, factor=.5, noise=noise, random_state=42)
    elif dataset is 'moons':
        X, Y = datasets.make_moons(n_samples=n_samples, noise=noise, random_state=42)
    elif dataset == 'xor':
        np.random.seed(42)
        step = int(n_samples/4)

        X = np.zeros((n_samples, 2))
        Y = np.zeros(n_samples)

        X[0*step:1*step, :] = noise * np.random.randn(step, 2)
        Y[0*step:1*step] = 1
        X[1*step:2*step, :] = np.array([1, 1]) + noise * np.random.randn(step, 2)
        Y[1*step:2*step] = 1

        X[2*step:3*step, :] = np.array([0, 1]) + noise * np.random.randn(step, 2)
        Y[2*step:3*step] = -1
        X[3*step:4*step, :] = np.array([1, 0]) + noise * np.random.randn(step, 2)
        Y[3*step:4*step] = -1

    elif dataset == 'periodic':
        np.random.seed(42)
        step = int(n_samples/4)

        X = np.zeros((n_samples, 2))
        Y = np.zeros(n_samples)

        X[0*step:1*step, :] = noise * np.random.randn(step, 2)
        Y[0*step:1*step] = 1
        X[1*step:2*step, :] = np.array([0, 2]) + noise * np.random.randn(step, 2)
        Y[1*step:2*step] = 1

        X[2*step:3*step, :] = np.array([0, 1]) + noise * np.random.randn(step, 2)
        Y[2*step:3*step] = -1
        X[3*step:4*step, :] = np.array([0, 3]) + noise * np.random.randn(step, 2)
        Y[3*step:4*step] = -1

    X = X[Y <= 1, :]
    Y = Y[Y <=1 ]
    Y[Y==0] = -1

    # Add the 1 feature.
    X = np.concatenate((X, np.ones((X.shape[0], 1))), axis=1)
    plot_support = True
    if kernel == 'poly':
        gamma = 1
        coef0 = 0
    elif kernel == 'sigmoid':
        gamma = np.power(10., bw)
        coef0 = 0
    elif kernel == 'rbf':
        gamma = np.power(10., -bw)
        coef0 = 0
    elif kernel == 'laplacian':
        gamma = np.power(10., -bw)
        coef0 = 0
        kernel = lambda X, Y: laplacian_kernel(X, Y, gamma)
        plot_support = False

    classifier = svm.SVC(kernel=kernel, C=np.power(10., -reg), gamma=gamma, degree=deg, coef0=coef0, tol=tol)
    classifier.fit(X, Y)

    # plot the line, the points, and the nearest vectors to the plane
    plt.figure()
    plt.clf()
    fig = plt.axes()
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-'}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

    if plot_support:
        plt.scatter(classifier.support_vectors_[:, 0], classifier.support_vectors_[:, 1], s=80,
                    facecolors='none', edgecolors='k')

    mins = np.min(X, 0)
    maxs = np.max(X, 0)
    x_min = mins[0] - 1
    x_max = maxs[0] + 1
    y_min = mins[1] - 1
    y_max = maxs[1] + 1

    XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]
    Xtest = np.c_[XX.ravel(), YY.ravel(), np.ones_like(XX.ravel())]
    # Xtest = np.c_[XX.ravel(), YY.ravel()]
    Z = classifier.decision_function(Xtest)

    # Put the result into a color plot
    Z = Z.reshape(XX.shape)
    # plt.figure(fignum, figsize=(4, 3))
    plt.contourf(XX, YY, Z > 0, cmap=plt.cm.jet, alpha=0.3)
    plt.contour(XX, YY, Z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'], levels=[-.99, 0, .99])

    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)

#     plt.xticks(())
#     plt.yticks(())

interact_manual(svm_kernel,
        dataset=['blobs', 'circles', 'moons', 'xor', 'periodic'],
        kernel=['poly', 'rbf', 'laplacian'],
        reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
        bw=ipywidgets.FloatSlider(value=-1,
                                    min=-3,
                                    max=3,
                                    step=0.1,
                                    readout_format='.1f',
                                    description='Bandwidth 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
        deg=ipywidgets.IntSlider(
                         value=1,
                         min=1,
                         max=10,
                         step=1,
                         description='Degree of Poly:',
                         style={'description_width': 'initial'}),
        noise=ipywidgets.FloatSlider(value=0.05,
                                    min=0.01,
                                    max=0.3,
                                    step=0.01,
                                    readout_format='.2f',
                                    description='Noise level:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
        );
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

## KNN

The K-Nearest Neighbors classifier is very easy. The label of a next sample is the label most voted by the $k$ training samples that are closer to this sample.

A simple implementation is $$\hat{y} = \text{sign}\left\{\sum_{i \in \mathcal{N}_k (x)} K(x_i, x) y_i\right\},$$

where $\mathcal{N}_k (x)$ is the set with the $k$ closest neighbours of $x$, $K(x_i, x)$ is a weighting coefficient, and $y_i$ is the label of example $i$. Usually $K(\cdot, \cdot)$ can be a kernel function, that measures the similarity between two points. In the vanilla k-NN method the kernel is just the identity function.

```python
from classifiers import kNN

noise = 0.4
X, Y = generate_circular_separable_data(num_points, noise=noise, offset=1)

def change_k_nn(k):
    classifier = kNN(X, Y, k)
    fig = plt.subplot(111)
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-'}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

    opt = {'n_points': 20, 'x_label': '$x$', 'y_label': '$y$'}
    plot_helpers.plot_classification_boundaries(X, classifier, fig=fig, options=opt)

interact(change_k_nn,
         k=ipywidgets.IntSlider(value=1,
                                min=1,
                                max=9,
                                step=2,
                                description='k',
                                style={'description_width': 'initial'},
                                continuous_update=False)
         );

```

# Cost Sensitivity

When there are more positive than negative examples the then, to overcome this uneven distribution of data, the cost of mislabeling can be changed between classes.

The cost can be expressed as:  $$ w^\star = \arg \min_w \sum_{i=1}^{N} c(y_i) \left[y_i\neq \text{sign} (w^\top x_i) \right], $$
where $c(y_i)$ is a cost that depends on the class.

```python
num_positive = 20  # Number of points per class
num_negative = 100  # Number of points per class
noise = 0.3  # Noise Level (needed for data generation).

X, Y = generate_linear_separable_data(num_positive, num_negative, noise=noise, dim=2)
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC

def cost_sensitivity(algorithm, ratio):
    class_weight = {-1: 1, 1:np.power(10., ratio)}
    if algorithm == 'SVM':
        classifier = LinearSVC(class_weight=class_weight)
    elif algorithm == 'Perceptron':
        classifier = SGDClassifier(loss='perceptron', random_state=1, class_weight=class_weight)

    classifier.fit(X[:,:2], Y)

    X0, X1 = X[:, 0], X[:, 1]
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    h = .02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    fig = plt.subplot(111)
    plot_helpers.plot_contours(fig, classifier, xx, yy, cmap=plt.cm.jet, alpha=0.3)
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-'}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

interact(cost_sensitivity,
         algorithm=ipywidgets.RadioButtons(
             options=['SVM', 'Perceptron'],
             value='SVM',
             description='Algorithm:',
             style={'description_width': 'initial'}),
         ratio=ipywidgets.FloatSlider(
             value=0,
             min=-1,
             max=3,
             step=0.5,
             description='Cost Ratio 10^:',
             style={'description_width': 'initial'},
             continuous_update=False)
         );
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

# L1 Regularization

The L-1 regularization method uses a regularizer of the form $R(w) = ||w||_1$ which is non-differentiable. However, the subgradient exits and is:

$$\partial(||w|||_1) = \left\{\begin{array}{cc} \text{sign} (w)& \text{if } w \neq 0 \\ [-1, 1]  & \text{if } w = 0 \end{array} \right. $$

This regularization method penalizes weights and induces sparsity in the solutions. That is, most of the entries of the solution $w^\star$ will be zero.

```python
num_positive = 50  # Number of points per class
num_negative = 50  # Number of points per class

noise = 0.3  # Noise Level (needed for data generation).

X, Y = generate_linear_separable_data(num_positive, num_negative, offset=np.array([1, .2]), noise=noise, dim=2)
X = X - np.mean(X, axis=0)

from sklearn.linear_model import SGDClassifier

def regularization(regularizer, reg):
    np.random.seed(42)
    classifier = SGDClassifier(loss='perceptron', penalty=regularizer, alpha = np.power(10., reg), random_state=1)
    classifier.fit(X[:,:2], Y)

    X0, X1 = X[:, 0], X[:, 1]
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    h = .02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    fig = plt.subplot(111)
    contour = plot_helpers.plot_contours(fig, classifier, xx, yy, cmap=plt.cm.jet, alpha=0.3)
    plt.colorbar(contour)
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-', 'legend': True}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

interact(regularization,
         regularizer=ipywidgets.RadioButtons(
             options=['l1', 'l2'],
             value='l1',
             description='Algorithm:',
             style={'description_width': 'initial'}),
         reg=ipywidgets.FloatSlider(
             value=-3,
             min=-3,
             max=0,
             step=0.5,
             description='Regularizer 10^:',
             style={'description_width': 'initial'},
             continuous_update=False)
         );
```

# Multiclass Classification

In multiclass classification there are two strategies: OvO (One vs One) or OvR (One vs The Rest).

```python
from sklearn.multiclass import OneVsOneClassifier, OneVsRestClassifier
from sklearn import datasets
from sklearn.svm import LinearSVC
from sklearn.linear_model import SGDClassifier

np.random.seed(1)
iris = datasets.load_iris()
X, y = iris.data[:,:2], iris.target
mean = X.mean(axis=0)
std = X.std(axis=0)
X = (X - mean) / std

def multiclass(strategy):
    if strategy == 'OvO':
        classifier =  OneVsOneClassifier(SGDClassifier(loss='perceptron', alpha=0.001, n_iter=100, random_state=0))
#         classifier =  OneVsOneClassifier(LinearSVC(random_state=0))
        colors = ['g', 'r', 'b']
    elif strategy == 'OvR' or strategy == 'OvA':
        classifier =  OneVsRestClassifier(SGDClassifier(loss='perceptron', alpha=0.001, n_iter=100, random_state=0))
#         classifier =  OneVsRestClassifier(LinearSVC(random_state=0))
        colors = ['b', 'g', 'r']

    classifier.fit(X, y)
    X0, X1 = X[:, 0], X[:, 1]
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    h = .02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    fig = plt.subplot(111)
    plot_helpers.plot_contours(fig, classifier, xx, yy, cmap=plt.cm.jet, alpha=0.3)

    opt = {'marker': 'bs', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8}
    plot_helpers.plot_data(X[np.where(y == 0)[0], 0], X[np.where(y == 0)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'g*', 'label': '+', 'size': 8}
    plot_helpers.plot_data(X[np.where(y == 1)[0], 0], X[np.where(y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'ro', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8, 'legend': True}
    plot_helpers.plot_data(X[np.where(y == 2)[0], 0], X[np.where(y == 2)[0], 1], fig=fig, options=opt)

    def plot_hyperplane(clf, color):
        coef = clf.coef_
        intercept = clf.intercept_
        def line(x0):
            return (-(x0 * coef[0, 0]) - intercept[0]) / coef[0, 1]

        plt.plot([x_min, x_max], [line(x_min), line(x_max)],
                 ls="--", color=color)

    for estimator, color in zip(classifier.estimators_, colors):
        plot_hyperplane(estimator, color)

    fig.set_xlim([x_min, x_max])
    fig.set_ylim([y_min, y_max]);

interact(multiclass, strategy=['OvO', 'OvA']);
```

# Multiclass with Nearest Neighbours

```python
from sklearn import neighbors

np.random.seed(1)
iris = datasets.load_iris()
X, y = iris.data[:,:2], iris.target

def change_n_neighbors(n_neighbors):
    for weights in ['uniform']:#, 'distance']:
        # we create an instance of Neighbours Classifier and fit the data.
        clf = neighbors.KNeighborsClassifier(n_neighbors, weights=weights)
        clf.fit(X, y)

        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        h = 0.1
        x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
        y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
        xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                             np.arange(y_min, y_max, h))
        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        plt.figure()
        fig = plt.axes()
        plt.contourf(xx, yy, Z, cmap=plt.cm.jet, alpha=0.3)

        # Plot also the training points
        opt = {'marker': 'bs', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8}
        plot_helpers.plot_data(X[np.where(y == 0)[0], 0], X[np.where(y == 0)[0], 1], fig=fig, options=opt)
        opt = {'marker': 'g*', 'label': '+', 'size': 8}
        plot_helpers.plot_data(X[np.where(y == 1)[0], 0], X[np.where(y == 1)[0], 1], fig=fig, options=opt)
        opt = {'marker': 'ro', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8, 'legend': True}
        plot_helpers.plot_data(X[np.where(y == 2)[0], 0], X[np.where(y == 2)[0], 1], fig=fig, options=opt)

        plt.xlim(xx.min(), xx.max())
        plt.ylim(yy.min(), yy.max())
        plt.title("3-Class classification (k = %i, weights = '%s')"
                  % (n_neighbors, weights))

interact(change_n_neighbors, n_neighbors=ipywidgets.IntSlider(value=1, min=1, max=15, step=2));
```

```python

```