# Rapid adiabatic passage in a 2-level system
This notebook illustrates using the `toy_systems` package to set up a Hamiltonian and decay matrix for a two level system and then performing rapid adiabatic passage on the system. The main difference between this notebook and the notebook on Rabi oscillations (*Rabi oscillations in 2-level system.ipynb*) is that the Rabi rate and detuning of the of the drive field will be time-dependent. Thus this notebook primarily serves to illustrate how to deal with a time-dependent Hamiltonian.

## Imports
Start by importing the necessary packages

```python
%load_ext autoreload
%autoreload 2

import joblib
import matplotlib.pyplot as plt
from matplotlib import animation
from mpl_toolkits.mplot3d import Axes3D
plt.style.use("ggplot")
import numpy as np
import qutip
from sympy import Symbol

from toy_systems.couplings import ToyCoupling, ToyEnergy
from toy_systems.decays import ToyDecay
from toy_systems.hamiltonian import Hamiltonian
from toy_systems.quantum_system import QuantumSystem
from toy_systems.states import Basis, BasisState, ToyQuantumNumbers
```

## Define states
We start by defining the two states of the system: the ground state $|g\rangle$ and the excited $|e\rangle$, which we will later set to have a decay path to $|g\rangle$:

```python
g = BasisState(qn=ToyQuantumNumbers(label="g"))
e = BasisState(qn=ToyQuantumNumbers(label="e"))

# Print the states as a check
print(f"|g> = {g}")
print(f"|e> = {e}")

# Check that the states are orthogonal by taking inner product
print(f"<e|g> = {e @ g}")
```

    |g> = |g>
    |e> = |e>
    <e|g> = 0

## Define basis
We use the two orthogonal states to define a basis as shown below. The indexing of the matrix and vector representations of operators is determined by the order of the states when generating the state

```python
basis = Basis((g,e))
basis.print()
```

    |0> = |g>
    |1> = |e>

## Define energies
I'm going to define the system in the rotating frame so that the Hamiltonian doesn't have any quickly rotating terms of the form $e^{i\omega t}$. The detuning between the driving field and the natural frequency of the system is taken to be $\Delta(t)$ (time-varying), so I can set the energies of the two states to be $E_g = -\Delta/2$ and $E_e = \Delta/2$.

Note: the time dependence will be multiplied by the value of $\Delta$ when time-evolving the Hamiltonian.

```python
Δ = Symbol('Delta') # Using sympy to generate a symbolic Hamiltonian first (to make a "Δ" type "\Delta", then press tab)
E_g = ToyEnergy([g], -Δ/2, time_dep = "t")
E_e = ToyEnergy([e], +Δ/2, time_dep = "t") # Note the linear time-dependence
```

```python
E_g.time_dep
```

    't'

### Define coupling
I'm taking the strength of the driving field to vary as a Gaussian over time. This time dependence of the drive field may result in the probability of state transfer durin RAP being slightly different from

```python
Ω = Symbol('Omega')
coupling = ToyCoupling(g, e, Ω, time_dep = "exp(-t**2/(2*sigma**2))", time_args={"sigma": 1})
```

## Define Hamiltonian
Now that we have the states, energies and couplings, we can define the Hamiltonian

```python
H = Hamiltonian((E_g, E_e, coupling), basis)
display(H)
```

    H =
    [
    [-0.5, 1.0],
    [ 1.0, 0.5]]

    in basis Basis: name = None

## Define decay matrix from |e> to |g>
Can add a spontaneous decay from the excited state to the ground state as shown below:

```python
decay = ToyDecay(e, g, Symbol("Gamma"))
```

## Define a QuantumSystem
The QuantumSystem object combines the basis, Hamiltonian and decays to make setting parameters for time evolution using QuTiP more convenient.

```python
# Define the system
system = QuantumSystem(
    basis=basis,
    couplings=[E_g, E_e, coupling],
    decays=[decay],
)

# Get representations of the Hamiltonian and the decays that will be accepted by qutip
Hqobj, c_qobj = system.get_qobjs()
```

## Time-evolution using `QuTiP`
The `toy_systems` package is designed to take care of generating the Hamiltonian and collapse operators for the quantum system, and spit out a `qutip.QobjEVo`-object. The actual time-evolution is then taken care of by `qutip` (or something else if the user so desires).

To get the adiabatic passage to work, the adiabatic condition needs to be fulfilled: $\Omega^2/\frac{dD}{dt} >> 1$. The width of the Gaussian describing the time dependence of the drive field also seems to play a role.

```python
# Get a pointer to the time-evolution arguments
args = Hqobj.args
print("Keys for setting arguments:")
print(f"args = {args}")
```

    Keys for setting arguments:
    args = {'Delta': 1, 'sigma': 1, 'Omega': 1, 'Gamma': 1}

```python
# Generate a Qobj representing the initial state
f=.99
psi0 = (np.sqrt(f)*g+np.sqrt(1-f)*e).qobj(basis)

# Make operators for getting the probability of being in each state
P_g_op = qutip.Qobj((1*g).density_matrix(basis), type = "oper")
P_e_op = qutip.Qobj((1*e).density_matrix(basis), type = "oper")

# Pauli matrices for calculating components of Bloch vector
sx = qutip.sigmax()
sy = qutip.sigmay()
sz = qutip.sigmaz()

# Set the parameters for the system
args["Delta"] = -1e-1 # This is actually the rate at which delta varies, i.e. dD/dt
args["sigma"] = 1/np.abs(args["Delta"]) # Width of the Gaussian describing time-dependence of drive field
args["Omega"] = 1 # The maximum value of the drive field Rabi rate
args["Gamma"] = 0

# Times at which result is requested
times = np.linspace(-5/np.abs(args["Delta"]), 5/np.abs(args["Delta"]), 1001)

# The solver used by qutip sometimes doesn't seem to notice that adiabatic passage is supposed to be happening, so reduce max step size to
# mitigate this
options = qutip.solver.Options(method = 'adams', nsteps=100000, max_step=1e0/np.abs(args["Delta"]),)

# Setup a progress bar
pb = qutip.ui.progressbar.EnhancedTextProgressBar()

# Run the time-evolution
result = qutip.mesolve(Hqobj, psi0, times, c_ops = c_qobj, e_ops = [P_g_op, P_e_op, sx, sy, sz],
                      progress_bar=pb, options = options)
```

     Total run time:   0.89s*] Elapsed 0.89s / Remaining 00:00:00:00[*********79%******     ] Elapsed 0.68s / Remaining 00:00:00:00

Plot the result:

```python
fig, (ax,ax2) = plt.subplots(2,1, figsize = (16,9))
ax.plot(times, result.expect[0], label = "P_g")
ax.plot(times, result.expect[1], label = "P_e")
ax.legend()
ax.set_xlabel("Time / (1/Ω)")
ax.set_ylabel("Population in each state")

# Plot the coupling strength and detuning also
ax2c = ax2.twinx()
# ax2.grid(False)
# ax2c.grid(False)
ln1 = ax2.plot(times, E_e.eval(times, args), label = "Δ")
ln2 = coupling.plot_time_dep(times, args, ax = ax2c, label = 'Ω', c = 'tab:blue')
ax2c.grid(False)
ax2c.set_ylabel('')
ax2.legend(ln1+ln2, [l.get_label() for l in (ln1 + ln2)])

```

### Visualizing on the Bloch sphere

```python
# Calculate components of Bloch vector
u = -result.expect[2]
v = result.expect[3]
w = result.expect[4]

# Calculate detuning and coupling strength
deltas = E_e.eval(times, time_args=args)
Omegas = coupling.eval(times, time_args=args)
W_length = np.sqrt(deltas**2 + 4*Omegas**2)

# Set up the bloch sphere
fig = plt.figure(figsize = (16, 9))
ax0 = fig.add_subplot(121, projection = '3d')
ax0.set_title('Bloch sphere', fontsize = 18)
b = qutip.Bloch(fig = fig, axes = ax0)
b.frame_width = 1
b.figsize = (16,16)

# Add initial Bloch vector and Rabi vector
b.add_vectors([np.array([u[0],v[0],w[0]]), np.array([Omegas[0], 0, deltas[0]])])
b.make_sphere()

# Set up plot of detuning
ax1 = fig.add_subplot(222)
ax1.plot(times, deltas, label = "Δ")
ax1.set_ylabel('Detuning', fontsize = 16)
ax1.plot(times[0], deltas[0], marker = 'o', markersize=8, c = 'k', fillstyle='none')

# Set up plot of Rabi rate
ax2 = fig.add_subplot(224)
ax2.plot(times, Omegas, label = "Ω", c = 'C1')
ax2.set_ylabel('Rabi rate', fontsize = 16)
ax2.plot(times[0], Omegas[0], marker = 'o', markersize=8, c = 'k', fillstyle='none')

# Define function that generates frames for animation
def animate(i):
    b.clear()
    b.add_vectors([np.array([u[i],v[i],w[i]]), np.array([2*Omegas[i], 0, deltas[i]])/W_length[i]])
    b.add_points([u[:i+1],v[:i+1],w[:i+1]])
    b.make_sphere()

    ax1.clear()
    ax1.plot(times, deltas, label = "Δ")
    ax1.set_ylabel('Detuning', fontsize = 16)
    ax1.plot(times[i], deltas[i], marker = 'o', markersize=8, c = 'k', fillstyle='none')

    ax2.clear()
    ax2.plot(times, Omegas, label = "Ω", c = 'C1')
    ax2.set_ylabel('Rabi rate', fontsize = 16)
    ax2.plot(times[i], Omegas[i], marker = 'o', markersize=8, c = 'k', fillstyle='none')
    ax2.set_xlabel('Time')

    return fig, (ax0, ax1, ax2)

```

```python
fig, _ = animate(350)
fig
```

```python
ani = animation.FuncAnimation(fig, animate, range(len(times)), blit=False, repeat=False, )
# ani.save('adiabatic_passage_f100.mp4', fps = 60)
```

### Visualizing v2

```python
# Calculate components of Bloch vector
u = -result.expect[2]
v = result.expect[3]
w = result.expect[4]

# Calculate detuning and coupling strength
deltas = E_e.eval(times, time_args=args)
Omegas = 2*coupling.eval(times, time_args=args)
W_length = np.sqrt(deltas**2 + Omegas**2)

# Set up the bloch sphere
fig = plt.figure(figsize = (16, 9))
ax0 = fig.add_subplot(221, projection = '3d')
ax0.set_title('Bloch sphere', fontsize = 18)
b = qutip.Bloch(fig = fig, axes = ax0)
b.frame_width = 1
b.figsize = (16,16)

# Add initial Bloch vector and Rabi vector
b.add_vectors([np.array([u[0],v[0],w[0]]), np.array([Omegas[0], 0, deltas[0]])])
b.make_sphere()

# Set up plot of detuning
ax1 = fig.add_subplot(222)
ax1.plot(times, deltas, label = "Δ")
ax1.set_ylabel('Detuning', fontsize = 16)
ax1.plot(times[0], deltas[0], marker = 'o', markersize=8, c = 'k', fillstyle='none')

# Set up plot of Rabi rate
ax2 = fig.add_subplot(224)
ax2.plot(times, Omegas, label = "Ω", c = 'C1')
ax2.set_ylabel('Rabi rate', fontsize = 16)
ax2.set_xlabel('Time')
ax2.plot(times[0], Omegas[0], marker = 'o', markersize=8, c = 'k', fillstyle='none')

# Set up plot of populations
ax3 = fig.add_subplot(223)
ax3.plot(times, result.expect[0], label = r"P$_0$")
ax3.plot(times, result.expect[1], label = r"P$_1$")
ax3.set_ylabel('Population in each state', fontsize = 16)
ax3.axvline(times[0], ls = '--', c = 'k')
ax3.set_xlabel('Time')
ax3.legend()

# Define function that generates frames for animation
def animate(i):
    b.clear()
    b.add_vectors([np.array([u[i],v[i],w[i]]), np.array([Omegas[i], 0, deltas[i]])/W_length[i]])
    b.add_points([u[:i+1],v[:i+1],w[:i+1]])
    b.make_sphere()

    ax1.clear()
    ax1.plot(times, deltas, label = "Δ")
    ax1.set_ylabel('Detuning', fontsize = 16)
    ax1.plot(times[i], deltas[i], marker = 'o', markersize=8, c = 'k', fillstyle='none')

    ax2.clear()
    ax2.plot(times, Omegas, label = "Ω", c = 'C1')
    ax2.set_ylabel('Rabi rate', fontsize = 16)
    ax2.plot(times[i], Omegas[i], marker = 'o', markersize=8, c = 'k', fillstyle='none')
    ax2.set_xlabel('Time')

    ax3.clear()
    ax3.plot(times, result.expect[0], label = r"P$_0$")
    ax3.plot(times, result.expect[1], label = r"P$_1$")
    ax3.set_ylabel('Population in each state', fontsize = 16)
    ax3.axvline(times[i], ls = '--', c = 'k')
    ax3.set_xlabel('Time')
    ax3.legend()

    return fig, (ax0, ax1, ax2, ax3)
```

```python
fig, _ = animate(400)
fig
```

```python
# %%timeit
ani = animation.FuncAnimation(fig, animate, range(len(times)), blit=False, repeat=False)
ani.save('adiabatic_passage_f099_v2_norm_length.mp4', fps = 60)
```

## What does rapid mean?
Rapid and adiabatic usually don't go well together. Adiabatic, as mentioned earlier, refers to satisfying the condition $\Omega^2/\frac{dD}{dt} >> 1$ which basically says the detuning should change slowly enough compared to the Rabi rate. The "rapid" in "Rapid adiabatic passage" actually refers to being quick compared to the lifetime of the excited state; if the excited state decays before the RAP is complete we don't get the desired population inversion.

```python
# Generate a Qobj representing the initial state
psi0 = (1*g).qobj(basis)

# Make operators for getting the probability of being in each state
P_g_op = qutip.Qobj((1*g).density_matrix(basis), type = "oper")
P_e_op = qutip.Qobj((1*e).density_matrix(basis), type = "oper")

# Set the parameters for the system
args["Delta"] = 1
args["sigma"] = 2
args["Omega"] = 1
args["Gamma"] = 1

# Times at which result is requested
times = np.linspace(-20/args["Delta"], 20/args["Delta"], 1001)

# The solver used by qutip sometimes doesn't seem to notice that adiabatic passage is supposed to be happening, so reduce max step size to
# mitigate this
options = qutip.solver.Options(method = 'adams', nsteps=10000, max_step=1e0)

# Setup a progress bar
pb = qutip.ui.progressbar.EnhancedTextProgressBar()

# Run the time-evolution
result = qutip.mesolve(Hqobj, psi0, times, c_ops = c_qobj, e_ops = [P_g_op, P_e_op],
                      progress_bar=pb, options = options)

# Plot the result
fig, ax = plt.subplots(figsize = (16,9))
ax.plot(times, result.expect[0], label = "P_g")
ax.plot(times, result.expect[1], label = "P_e")
ax.legend()
ax.set_xlabel("Time / (1/Ω)")
ax.set_ylabel("Population in each state")
```

## Testing if Landau-Zener formula applies
The famous Landau-Zener formula for the probability of adiabatic transitions assumes a constant coupling strength and a linear detuning ramp. Under these conditions the probability of staying in the initial state can be solved for exactly and is given by $P_{ini} = 1-\exp\left(-2\pi\Omega^2/\left(\frac{dD}{dt}\right)\right)$. We are not using a constant coupling strength here, but I would expect the transition probabilities to be very close to the LZ formula predictions since the part that really matters is where $\Delta \approx 0$ where the Rabi rate is also changing quite slowly.

To investigate, I will do a parameter scan with the detuning rate. This will also illustrate using QuTiP and parallelization.

```python
# Define the function we want to run in parallel to perform the parameter scan
def parallel_func(Δ: float):

    # Generate a Qobj representing the initial state
    psi0 = (1*g).qobj(basis)

    # Make operators for getting the probability of being in each state
    P_g_op = qutip.Qobj((1*g).density_matrix(basis), type = "oper")
    P_e_op = qutip.Qobj((1*e).density_matrix(basis), type = "oper")

    args = {}

    # Set the parameters for the system
    args["Delta"] = Δ # This is actually the rate at which delta varies, i.e. dD/dt
    args["sigma"] = 2/Δ # Width of the Gaussian describing time-dependence of drive field
    args["Omega"] = 1 # The maximum value of the drive field Rabi rate
    args["Gamma"] = 0

    # Times at which result is requested
    times = np.linspace(-20/args["Delta"], 20/args["Delta"], 1001)

    # The solver used by qutip sometimes doesn't seem to notice that adiabatic passage is supposed to be happening, so reduce max step size to
    # mitigate this
    options = qutip.solver.Options(method = 'adams', nsteps=100000, max_step=1e0/args["Delta"],rhs_reuse=True)

    # Run the time-evolution
    result = qutip.mesolve(Hqobj, psi0, times, args = args, c_ops = c_qobj, e_ops = [P_g_op, P_e_op], options = options, progress_bar=pb)

    # Plot the result
    fig, ax = plt.subplots(figsize = (16,9))
    ax.plot(times, result.expect[0], label = "P_g")
    ax.plot(times, result.expect[1], label = "P_e")
    ax.legend()
    ax.set_xlabel("Time / (1/Ω)")
    ax.set_ylabel("Population in each state")
    ax.set_title(f"Δ = {Δ}")

    return result.expect[0][-1]

# Define the values of dD/dt we want to try
Δs = 10**np.linspace(-0.5,2,30)

# Define a loop that will be excecuted in parallel using joblib
probs = (joblib.Parallel(n_jobs = 5, backend='loky', verbose = True)(joblib.delayed(parallel_func)(Δ) for Δ in Δs))
```

    [Parallel(n_jobs=5)]: Using backend LokyBackend with 5 concurrent workers.
    [Parallel(n_jobs=5)]: Done  30 out of  30 | elapsed:    1.7s finished

```python
# Plot the result
fig, (ax, ax2) = plt.subplots(2,1, figsize = (16,9))
ax.plot(np.log10(Δs), probs, 'x')
ax.set_xlabel("dΔ/dt")
ax.set_ylabel("P_g")
ax2.plot(1/Δs, np.log(np.array(probs)), 'x', label = 'Simulation')
ax2.set_xlabel("1/(dΔ/dt)")
ax2.set_ylabel("ln(P_g)")

# Fit a line to the log plot
from scipy.stats import linregress
x_fit = 1/Δs[3:-3]
y_fit = np.log(np.array(probs[3:-3]))
fit_result = linregress(x_fit, y_fit)
ax2.plot(x_fit, fit_result.slope*x_fit+fit_result.intercept)

print(f"slope: {fit_result.slope/(2*np.pi):.3f}*2*pi, intercept: {fit_result.intercept:.3f}")
```

The lower plot should be a line if the LZ formula applies exactly, with the gradient given by $k = -2\pi\Omega^2$. Clearly this is not exactly the case but the gradient for the linear fit is close to the expected value. The discrepanceies are presumably due to having a time dependent coupling strength.

```python
import matplotlib
matplotlib.__version__
```

    '3.5.0'

```python

```

examples/Rapid adiabatic passage in a 2-level system.ipynb


```python
import math
import numpy
from matplotlib import pyplot
```

# BEM method

```python
#Q = 2000/3    #strength of the source-sheet,stb/d
h=25.26        #thickness of local gridblock,ft
phi=0.2        #porosity
kx=200         #pemerability in x direction,md
ky=200         #pemerability in y direction,md
kr=kx/ky       #pemerability ratio
miu=1          #viscosity,cp

Nw=1           #Number of well
Qwell_1=2000   #Flow rate of well 1
Boundary_V=-400 #boundary velocity ft/day
```

## Boundary Discretization
we will create a discretization of the body geometry into panels (line segments in 2D). A panel's attributes are: its starting point, end point and mid-point, its length and its orientation. See the following figure for the nomenclature used in the code and equations below.

Figure 1. Nomenclature of the boundary element in the local coordinates
### Create panel and well class

```python
class Panel:
    """Contains information related to a panel."""
    def __init__(self, xa, ya, xb, yb):
        """Creates a panel.

        Arguments
        ---------
        xa, ya -- Cartesian coordinates of the first end-point.
        xb, yb -- Cartesian coordinates of the second end-point.
        """
        self.xa, self.ya = xa, ya
        self.xb, self.yb = xb, yb

        self.xc, self.yc = (xa+xb)/2, (ya+yb)/2       # control-point (center-point)
        self.length = math.sqrt((xb-xa)**2+(yb-ya)**2)     # length of the panel

        # orientation of the panel (angle between x-axis and panel)
        self.sinalpha=(yb-ya)/self.length
        self.cosalpha=(xb-xa)/self.length

        self.Q = 0.                                # source strength
        self.U = 0.                                # velocity component
        self.V = 0.                                # velocity component
        self.P = 0.                                # pressure coefficient

class Well:
    """Contains information related to a panel."""
    def __init__(self, xw, yw,rw,Q):
        """Creates a panel.

        Arguments
        ---------
        xw, yw -- Cartesian coordinates of well source.
        Q      -- Flow rate of well source.
        rw     -- radius of well source.
        """
        self.xw, self.yw = xw, yw

        self.Q = Q                               # source strength
        self.rw = rw                                # velocity component

```

We create a node distribution on the boundary that is refined near the corner with cosspace function

```python
def cosspace(st,ed,N):
    N=N+1
    AngleInc=numpy.pi/(N-1)
    CurAngle = AngleInc
    space=numpy.linspace(0,1,N)
    space[0]=st
    for i in range(N-1):
        space[i+1] = 0.5*numpy.abs(ed-st)*(1 - math.cos(CurAngle));
        CurAngle += AngleInc
    if ed<st:
        space[0]=ed
        space=space[::-1]
    return space
```

### Discretize boundary element along the boundary
Here we implement BEM in a squre grid

```python
N=80     #Number of boundary element
Nbd=20   #Number of boundary element in each boundary
Dx=1.    #Grid block length in X direction
Dy=1.    #Gird block lenght in Y direction

#Create the array
x_ends = numpy.linspace(0, Dx, N)    # computes a 1D-array for x
y_ends = numpy.linspace(0, Dy, N)    # computes a 1D-array for y
interval=cosspace(0,Dx,Nbd)
rinterval=cosspace(Dx,0,Nbd)
#interval=numpy.linspace(0,1,Nbd+1)
#rinterval=numpy.linspace(1,0,Nbd+1)

#Define the rectangle boundary

for i in range(Nbd):
    x_ends[i]=0
    y_ends[i]=interval[i]

for i in range(Nbd):
    x_ends[i+Nbd]=interval[i]
    y_ends[i+Nbd]=Dy

for i in range(Nbd):
    x_ends[i+Nbd*2]=Dx
    y_ends[i+Nbd*2]=rinterval[i]

for i in range(Nbd):
    x_ends[i+Nbd*3]=rinterval[i]
    y_ends[i+Nbd*3]=0

x_ends,y_ends=numpy.append(x_ends, x_ends[0]), numpy.append(y_ends, y_ends[0])

#Define the panel
panels = numpy.empty(N, dtype=object)
for i in range(N):
    panels[i] = Panel(x_ends[i], y_ends[i], x_ends[i+1], y_ends[i+1])

#Define the well
wells = numpy.empty(Nw, dtype=object)

wells[0]=Well(Dx/2,Dy/2,0.025,Qwell_1)
```

```python
#for i in range(N):
 #   print("Panel Coordinate (%s,%s) sina,cosa (%s,%s) " % (panels[i].xc,panels[i].yc,panels[i].sinalpha,panels[i].cosalpha))
#print("Well Location (%s,%s) radius: %s Flow rate:%s  " % (wells[0].xw,wells[0].yw,wells[0].rw,wells[0].Q))
```

### Plot boundary elements and wells

```python
#Plot the panel
%matplotlib inline

val_x, val_y = 0.3, 0.3
x_min, x_max = min(panel.xa for panel in panels), max(panel.xa for panel in panels)
y_min, y_max = min(panel.ya for panel in panels), max(panel.ya for panel in panels)
x_start, x_end = x_min-val_x*(x_max-x_min), x_max+val_x*(x_max-x_min)
y_start, y_end = y_min-val_y*(y_max-y_min), y_max+val_y*(y_max-y_min)

size = 5
pyplot.figure(figsize=(size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)

pyplot.plot(numpy.append([panel.xa for panel in panels], panels[0].xa),
         numpy.append([panel.ya for panel in panels], panels[0].ya),
         linestyle='-', linewidth=1, marker='o', markersize=6, color='#CD2305');
pyplot.scatter(wells[0].xw,wells[0].yw,s=100,alpha=0.5)

pyplot.legend(['panels', 'Wells'],
           loc=1, prop={'size':12})
```

## Boundary element implementation

Figure 2. Representation of a local gridblock with boundary elements

Generally, the influence of all the j panels on the i BE node can be expressed as follows:

\begin{matrix}
{{c}_{ij}}{{p}_{i}}+{{p}_{i}}\int_{{{s}_{j}}}{{{H}_{ij}}d{{s}_{j}}}=({{v}_{i}}\cdot \mathbf{n})\int_{{{s}_{j}}}{{{G}_{ij}}}d{{s}_{j}}
\end{matrix}
Where,

${{c}_{ij}}$  is the free term, cased by source position.

${{c}_{ij}}=\left\{ \begin{matrix}
   \begin{matrix}
   1 & \text{source j on the internal domain}  \\
\end{matrix}  \\
   \begin{matrix}
   0.5 & \text{source j on the boundary}  \\
\end{matrix}  \\
   \begin{matrix}
   0 & \text{source j on the external domain}  \\
\end{matrix}  \\
\end{matrix} \right.$

$\int_{{{s}_{j}}}{{{H}_{ij}}d{{s}_{j}}\text{ }}$ is the integrated effect of the boundary element source i on the resulting normal flux at BE node j.

$\int_{{{s}_{j}}}{{{G}_{ij}}}d{{s}_{j}}$ is the is the integrated effect of the boundary element source i on the resulting pressure at BE node j

### Line segment source solution for pressure and velocity (Derived recently)

The integrated effect can be formulated using line segment source solution, which givs:

\begin{equation}
\int_{{{s}_{j}}}{{{G}_{ij}}}d{{s}_{j}}=B{{Q}_{w}}=P({{{x}'}_{i}},{{{y}'}_{i}})=-\frac{70.60\mu }{h\sqrt{{{k}_{x}}{{k}_{y}}}}\int_{t=0}^{t={{l}_{j}}}{\ln \left\{ {{({x}'-t\cos {{\alpha }_{j}})}^{2}}+\frac{{{k}_{x}}}{{{k}_{y}}}{{({y}'-t\sin {{\alpha }_{j}})}^{2}} \right\}dt}\cdot {{Q}_{w}}
\end{equation}

\begin{equation}
\int_{{{s}_{j}}}{{{H}_{ij}}d{{s}_{j}}\text{ }}={{v}_{i}}(s)\cdot {{\mathbf{n}}_{i}}=-{{u}_{i}}\sin {{\alpha }_{i}}+{{v}_{i}}\cos {{\alpha }_{i}}
\end{equation}

Where,

\begin{equation}
u\left( {{{{x}'}}_{i}},{{{{y}'}}_{i}} \right)={{A}_{u}}{{Q}_{j}}=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}_{x}}}{{{k}_{y}}}}\int_{t=0}^{t={{l}_{j}}}{\frac{{{{{x}'}}_{i}}-t\cos {{\alpha }_{j}}}{{{\left( {{{{x}'}}_{i}}-t\cos {{\alpha }_{j}} \right)}^{2}}+\frac{{{k}_{x}}}{{{k}_{y}}}{{({{{{y}'}}_{i}}-t\sin {{\alpha }_{j}})}^{2}}}dt}\cdot {{Q}_{j}}
\end{equation}

\begin{equation}
v\left( {{{{x}'}}_{i}},{{{{y}'}}_{i}} \right)={{A}_{v}}{{Q}_{j}}=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}_{x}}}{{{k}_{y}}}}\int_{t=0}^{t={{l}_{j}}}{\frac{{{{{y}'}}_{i}}-t\sin {{\alpha }_{j}}}{{{\left( {{{{x}'}}_{i}}-t\cos {{\alpha }_{j}} \right)}^{2}}+\frac{{{k}_{x}}}{{{k}_{y}}}{{({{{{y}'}}_{i}}-t\sin {{\alpha }_{j}})}^{2}}}dt}\cdot {{Q}_{j}}
\end{equation}

## Line segment source Integration function (Bij and Aij)

```python
#Panel infuence factor Bij
def InflueceP(x, y, panel):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
#Transfer global coordinate point(x,y) to local coordinate
    x=x-panel.xa
    y=y-panel.ya
    L1=panel.length

#Calculate the pressure and velocity influence factor
    a=panel.cosalpha**2+kr*panel.sinalpha**2
    b=x*panel.cosalpha+kr*panel.sinalpha*y
    c=y*panel.cosalpha-x*panel.sinalpha
    dp=70.6*miu/h/math.sqrt(kx*ky)
    Cp = dp/a*(
                             (
                              b*math.log(x**2-2*b*L1+a*L1**2+kr*y**2)
                             -L1*a*math.log((x-L1*panel.cosalpha)**2+kr*(y-L1*panel.sinalpha)**2)
                             +2*math.sqrt(kr)*c*math.atan((b-a*L1)/math.sqrt(kr)/c)
                             )
                             -
                             (
                               b*math.log(x**2+kr*y**2)
                               +2*math.sqrt(kr)*c*math.atan((b)/math.sqrt(kr)/c)
                             )
                )
    #debug
    #print("a: %s b:%s c:%s  " % (a,b,c))
    #angle=math.atan((b-a*L1)/math.sqrt(kr)/c)*180/numpy.pi
    #print("Magic angle:%s"% angle)
    return Cp

def InflueceU(x, y, panel):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
#Transfer global coordinate point(x,y) to local coordinate
    x=x-panel.xa
    y=y-panel.ya
    L1=panel.length

#Calculate the pressure and velocity influence factor
    a=panel.cosalpha**2+kr*panel.sinalpha**2
    b=x*panel.cosalpha+kr*panel.sinalpha*y
    c=y*panel.cosalpha-x*panel.sinalpha
    dv=-0.4468/h/phi*math.sqrt(kx/ky)
    Cu = dv/a*(
                             (
                            panel.cosalpha*math.log(x**2-2*b*L1+a*L1**2+kr*y**2)+ 2*math.sqrt(kr)*panel.sinalpha*math.atan((a*L1-b)/math.sqrt(kr)/c)
                             )
                            -
                             (
                            panel.cosalpha*math.log(x**2+kr*y**2)+2*math.sqrt(kr)*panel.sinalpha*math.atan((-b)/math.sqrt(kr)/c)
                             )
                     )
    #print("a: %s b:%s c:%s  " % (a,b,c))
    #angle=math.atan((b-a*L1)/math.sqrt(kr)/c)*180/numpy.pi
    #print("Magic angle:%s"% angle)
    return Cu

def InflueceV(x, y, panel):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
#Transfer global coordinate point(x,y) to local coordinate
    x=x-panel.xa
    y=y-panel.ya
    L1=panel.length

#Calculate the pressure and velocity influence factor
    a=panel.cosalpha**2+kr*panel.sinalpha**2
    b=x*panel.cosalpha+kr*panel.sinalpha*y
    c=y*panel.cosalpha-x*panel.sinalpha
    dv=-0.4468/h/phi*math.sqrt(kx/ky)
    Cv = dv/a*(
                             (
                            panel.sinalpha*math.log(x**2-2*b*L1+a*L1**2+kr*y**2)+ 2*math.sqrt(1/kr)*panel.cosalpha*math.atan((b-a*L1)/math.sqrt(kr)/c)
                             )
                            -
                             (
                            panel.sinalpha*math.log(x**2+kr*y**2)+2*math.sqrt(1/kr)*panel.cosalpha*math.atan((b)/math.sqrt(kr)/c)
                             )
                     )
    #print("a: %s b:%s c:%s  " % (a,b,c))
    #angle=math.atan((b-a*L1)/math.sqrt(kr)/c)*180/numpy.pi
    #print("Magic angle:%s"% angle)

    return Cv
```

## Well source function

### Line source solution for pressure and velocity (Datta-Gupta, 2007)

\begin{equation}
P(x,y)=B{{Q}_{w}}=-\frac{70.60\mu }{h\sqrt{{{k}_{x}}{{k}_{y}}}}\ln \left\{ {{(x-{{x}_{w}})}^{2}}+\frac{{{k}_{x}}}{{{k}_{y}}}{{(y-{{y}_{w}})}^{2}} \right\}{{Q}_{w}}+{{P}_{avg}}
\end{equation}

\begin{equation}
\frac{\partial P}{\partial x}=u=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}_{x}}}{{{k}_{y}}}}\sum\limits_{k=1}^{{{N}_{w}}}{{{Q}_{k}}}\frac{x-{{x}_{k}}}{{{\left( x-{{x}_{k}} \right)}^{2}}+\frac{{{k}_{x}}}{{{k}_{y}}}{{(y-{{y}_{k}})}^{2}}}
\end{equation}

\begin{equation}
\frac{\partial P}{\partial y}=v=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}_{x}}}{{{k}_{y}}}}\sum\limits_{k=1}^{{{N}_{w}}}{{{Q}_{k}}}\frac{y-{{y}_{k}}}{{{\left( x-{{x}_{k}} \right)}^{2}}+\frac{{{k}_{x}}}{{{k}_{y}}}{{(y-{{y}_{k}})}^{2}}}
\end{equation}

```python
#Well influence factor
def InflueceP_W(x, y, well):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    dp=-70.6*miu/h/math.sqrt(kx*ky)
    Cp=dp*math.log((x-well.xw)**2+kr*(y-well.yw)**2)
    return Cp

def InflueceU_W(x, y, well):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    dv=0.8936/h/phi*math.sqrt(kx/ky)
    Cu=dv*(x-well.xw)/((x-well.xw)**2+kr*(y-well.yw)**2)
    return Cu

def InflueceV_W(x, y, well):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    dv=0.8936/h/phi*math.sqrt(kx/ky)
    Cv=dv*(y-well.yw)/((x-well.xw)**2+kr*(y-well.yw)**2)
    return Cv
```

```python
#InflueceV(0.5,1,panels[3])
#InflueceP(0,0.5,panels[0])
#InflueceU(0,0.5,panels[0])
```

## BEM function solution
Generally, the influence of all the j panels on the i BE node can be expressed as follows:

\begin{matrix}
{{c}_{ij}}{{p}_{i}}+{{p}_{i}}\int_{{{s}_{j}}}{{{H}_{ij}}d{{s}_{j}}}=({{v}_{i}}\cdot \mathbf{n})\int_{{{s}_{j}}}{{{G}_{ij}}}d{{s}_{j}}
\end{matrix}

Applying boundary condition along the boundary on above equation, a linear systsem can be constructed as follows:

\begin{matrix}
\left[ {{{{H}'}}_{ij}} \right]\left[ {{P}_{i}} \right]=\left[ {{G}_{ij}} \right]\left[ {{v}_{i}}\cdot \mathbf{n} \right]
\end{matrix}

!!!!!MY IMPLEMENTATION MAY HAS SOME PROBLEM HERE!!!!!!

All the integration solution can be evaluated except on itself. Where,

$
\left[ {{{{H}'}}_{ij}} \right]=\left\{ \begin{matrix}
   \begin{matrix}
   {{H}_{ij}} & i\ne j  \\
\end{matrix}  \\
   \begin{matrix}
   {{H}_{ij}}+\frac{1}{2} & i=j  \\
\end{matrix}  \\
\end{matrix} \right.
$

Figure 3. Representation of coordinate systems and the principle of superstition with well source and boundary element source

As shown in Fig.3, the pressure and velocity at any point i in the local gridblock can be determined using Eqs. below. Applying principle of superposition for each BE node along the boundary (Fig. 3), boundary condition can be written as follows:

\begin{matrix}
   {{P}_{i}}(s)=\sum\limits_{j=1}^{M}{{{B}_{ij}}{{Q}_{j}}} & \text{constant pressure boundary}  \\
\end{matrix}

\begin{matrix}
   {{v}_{i}}(s)\cdot {{\mathbf{n}}_{i}}=\sum\limits_{j=1}^{M}{{{A}_{ij}}{{Q}_{j}}} & \text{constant flux boundary}  \\
\end{matrix}

The Pi and v ·n are the konwn boundary codition. The flow rate(strength) of boundary elements in Hij and Gij are the only unknown terms.
So we could rearrange the matrix above as linear system:

$
{{\left[ \begin{matrix}
   {{A}_{ij}}  \\
   {{B}_{ij}}  \\
\end{matrix} \right]}_{N\times N}}{{\left[ \begin{matrix}
   {{Q}_{j}}  \\
   {{Q}_{j}}  \\
\end{matrix} \right]}_{N\times 1}}={{\left[ \begin{matrix}
   -{{u}_{i}}\sin {{\alpha }_{i}}+{{v}_{i}}\cos {{\alpha }_{i}}  \\
   {{P}_{i}}  \\
\end{matrix} \right]}_{N\times 1}}
$

```python
def build_matrix(panels):
    """Builds the source matrix.

    Arguments
    ---------
    panels -- array of panels.

    Returns
    -------
    A -- NxN matrix (N is the number of panels).
    """
    N = len(panels)
    A = numpy.empty((N, N), dtype=float)
    #numpy.fill_diagonal(A, 0.5)

    for i, p_i in enumerate(panels): #target nodes
        for j, p_j in enumerate(panels): #BE source
            #if i != j:   ###Matrix construction
                if i>=0 and i<Nbd or i>=3*Nbd and i<4*Nbd:
                    A[i,j] = -p_j.sinalpha*InflueceU(p_i.xc, p_i.yc, p_j)+p_j.cosalpha*InflueceV(p_i.xc, p_i.yc, p_j)
                    #A[i,j] = InflueceP(p_i.xc, p_i.yc, p_j)
                if i>=Nbd and i<2*Nbd or i>=2*Nbd and i<3*Nbd:
                    A[i,j] = -p_j.sinalpha*InflueceU(p_i.xc, p_i.yc, p_j)+p_j.cosalpha*InflueceV(p_i.xc, p_i.yc, p_j)
                    #A[i,j] = InflueceP(p_i.xc, p_i.yc, p_j)

    return A

def build_rhs(panels):
    """Builds the RHS of the linear system.

    Arguments
    ---------
    panels -- array of panels.

    Returns
    -------
    b -- 1D array ((N+1)x1, N is the number of panels).
    """
    b = numpy.empty(len(panels), dtype=float)

    for i, panel in enumerate(panels):
        V_well=( -panel.sinalpha*Qwell_1*InflueceU_W(panel.xc, panel.yc, wells[0])+panel.cosalpha*Qwell_1*InflueceV_W(panel.xc, panel.yc, wells[0]) )
        if i>=0 and i<Nbd:
            b[i]=0+V_well
            #b[i]=4000
            #b[i]=84
        if i>=Nbd and i<2*Nbd:
            b[i]=-V_well
            #b[i]=-42
        if i>=2*Nbd and i<3*Nbd:
            b[i]=-V_well
            #b[i]=-42
        if i>=3*Nbd and i<4*Nbd:
            b[i]=0+V_well
            #b[i]=84
    return b
```

```python
#Qwell_1=300   #Flow rate of well 1
#Boundary_V=-227 #boundary velocity ft/day

A = build_matrix(panels)                    # computes the singularity matrix
b = build_rhs(panels)                       # computes the freestream RHS
```

    d:\Anaconda3\lib\site-packages\ipykernel\__main__.py:66: RuntimeWarning: divide by zero encountered in double_scalars
    d:\Anaconda3\lib\site-packages\ipykernel\__main__.py:70: RuntimeWarning: divide by zero encountered in double_scalars
    d:\Anaconda3\lib\site-packages\ipykernel\__main__.py:102: RuntimeWarning: divide by zero encountered in double_scalars
    d:\Anaconda3\lib\site-packages\ipykernel\__main__.py:106: RuntimeWarning: divide by zero encountered in double_scalars

```python
# solves the linear system
Q = numpy.linalg.solve(A, b)

for i, panel in enumerate(panels):
    panel.Q = Q[i]
```

# Plot results

```python
#Visulize the pressure and velocity field

#Define meshgrid
Nx, Ny = 50, 50                  # number of points in the x and y directions
x_start, x_end = -0.01, 1.01            # x-direction boundaries
y_start, y_end = -0.01, 1.01            # y-direction boundaries
x = numpy.linspace(x_start, x_end, Nx)    # computes a 1D-array for x
y = numpy.linspace(y_start, y_end, Ny)    # computes a 1D-array for y
X, Y = numpy.meshgrid(x, y)               # generates a mesh grid

#Calculate the velocity and pressure field
p = numpy.empty((Nx, Ny), dtype=float)
u = numpy.empty((Nx, Ny), dtype=float)
v = numpy.empty((Nx, Ny), dtype=float)

#for i, panel in enumerate(panels):
    #panel.Q = 0.

#panels[0].Q=100
#panels[5].Q=100
#Qwell_1=400

for i in range(Nx):
    for j in range(Ny):
        p[i,j] =sum([p.Q*InflueceP(X[i,j], Y[i,j], p) for p in panels])+Qwell_1*InflueceP_W(X[i,j], Y[i,j], wells[0])
        u[i,j] =sum([p.Q*InflueceU(X[i,j], Y[i,j], p) for p in panels])+Qwell_1*InflueceU_W(X[i,j], Y[i,j], wells[0])
        v[i,j] =sum([p.Q*InflueceV(X[i,j], Y[i,j], p) for p in panels])+Qwell_1*InflueceV_W(X[i,j], Y[i,j], wells[0])
        #p[i,j] =sum([p.Q*InflueceP(X[i,j], Y[i,j], p) for p in panels])
        #u[i,j] =sum([p.Q*InflueceU(X[i,j], Y[i,j], p) for p in panels])
        #v[i,j] =sum([p.Q*InflueceV(X[i,j], Y[i,j], p) for p in panels])
        #p[i,j] =Qwell_1*InflueceP_W(X[i,j], Y[i,j], wells[0])
        #u[i,j] =Qwell_1*InflueceU_W(X[i,j], Y[i,j], wells[0])
        #v[i,j] =Qwell_1*InflueceV_W(X[i,j], Y[i,j], wells[0])
```

```python
# plots the streamlines
%matplotlib inline

size = 6
pyplot.figure(figsize=(size, size))
pyplot.grid(True)
pyplot.title('Streamline field')
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(-0.2, 1.2)
pyplot.ylim(-0.2, 1.2)

pyplot.plot(numpy.append([panel.xa for panel in panels], panels[0].xa),
         numpy.append([panel.ya for panel in panels], panels[0].ya),
         linestyle='-', linewidth=1, marker='o', markersize=6, color='#CD2305');
stream =pyplot.streamplot(X, Y, u, v,density=2, linewidth=1, arrowsize=1, arrowstyle='->') #streamline
#cbar=pyplot.colorbar(orientation='vertical')

#equipotential=pyplot.contourf(X, Y, p1, extend='both')
```

```python
size = 7
pyplot.figure(figsize=(size, size-1))
pyplot.title('Pressure field')
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(0, 1)
pyplot.ylim(0, 1)

pyplot.contour(X, Y, p, 15, linewidths=0.5, colors='k')
pyplot.contourf(X, Y, p, 15, cmap='rainbow',
                  vmax=abs(p).max(), vmin=-abs(p).max())
pyplot.colorbar()  # draw colorbar
```

```python
size = 7
pyplot.figure(figsize=(size, size-1))
pyplot.title('Total Velocity field')
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(0, 1)
pyplot.ylim(0, 1)

Vtotal= numpy.sqrt(u**2+v**2)
#Vtotal= numpy.abs(v)
pyplot.contour(X, Y, Vtotal, 15, linewidths=0.5, colors='k')
pyplot.contourf(X, Y, Vtotal, 15, cmap='rainbow')
                  #vmax=50, vmin=0)
pyplot.colorbar()  # draw colorbar
```

```python
pyplot.title('Darcy velocity on the outflow boundary, x component (ft/day)')
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)

pyplot.plot(y, u[49,:], '--', linewidth=2)
pyplot.plot(9.8425+y, u[:,49], '--', linewidth=2)
u[:,49]
```

```python
pyplot.title('Darcy velocity on the outflow boundary, y component (ft/day)')

pyplot.plot(y, v[:,49], '--', linewidth=2)
pyplot.plot(9.8425+y, v[49,:], '--', linewidth=2)
v[49,:]
```

```python

```

```python

```


# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [ 1.33226763e-17  7.77156117e-17 -3.17107451e-17]
      stds:   [1. 1. 1.]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [1. 2. 3.]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029292401534124e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  0.0
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 2.16x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 4.05e-10
    b1 relative error: 5.33e-07
    b2 relative error: 6.66e-08
    b3 relative error: 1.01e-10
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 6.96e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 3.61e-08
    b2 relative error: 1.62e-06
    b3 relative error: 1.73e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 3.48e-09
    gamma1 relative error: 8.80e-09
    gamma2 relative error: 5.28e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340975
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.314000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.385000; val_acc: 0.279000
    (Iteration 41 / 200) loss: 2.041103
    (Epoch 3 / 10) train acc: 0.494000; val_acc: 0.308000
    (Iteration 61 / 200) loss: 1.753902
    (Epoch 4 / 10) train acc: 0.531000; val_acc: 0.307000
    (Iteration 81 / 200) loss: 1.246584
    (Epoch 5 / 10) train acc: 0.574000; val_acc: 0.313000
    (Iteration 101 / 200) loss: 1.320590
    (Epoch 6 / 10) train acc: 0.634000; val_acc: 0.338000
    (Iteration 121 / 200) loss: 1.157329
    (Epoch 7 / 10) train acc: 0.689000; val_acc: 0.325000
    (Iteration 141 / 200) loss: 1.145515
    (Epoch 8 / 10) train acc: 0.770000; val_acc: 0.331000
    (Iteration 161 / 200) loss: 0.671748
    (Epoch 9 / 10) train acc: 0.804000; val_acc: 0.335000
    (Iteration 181 / 200) loss: 0.906446
    (Epoch 10 / 10) train acc: 0.769000; val_acc: 0.322000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696059
    (Epoch 6 / 10) train acc: 0.535000; val_acc: 0.345000
    (Iteration 121 / 200) loss: 1.557987
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.304000
    (Iteration 141 / 200) loss: 1.432189
    (Epoch 8 / 10) train acc: 0.628000; val_acc: 0.339000
    (Iteration 161 / 200) loss: 1.033931
    (Epoch 9 / 10) train acc: 0.661000; val_acc: 0.340000
    (Iteration 181 / 200) loss: 0.901034
    (Epoch 10 / 10) train acc: 0.726000; val_acc: 0.318000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers_ws[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
Networks with batch normalization do not depend on correct initialization as much as the normal networks, because the gradient does not explode for them.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
For very small batch sizes batch normalization harms the learning process, because the batch is not representative enough for the whole dataset, and mean and variance of the batch differes too much from the total mean and variance.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
1. This one is like instance normalization, so it's not analogous to any of batch/layer normalization.
2. Analogous to Layer normalization.
3. Analogous to Batch normalization. (partially, does not have deviation 1)
4. Not analogous to anything.

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(N)
beta = np.zeros(N)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3. 3.] , beta= [5. 5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(N)
beta = np.random.randn(N)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  4.468136794434158e-09
    dgamma error:  2.0556513782477793e-12
    dbeta error:  1.3672276028472051e-11

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
1. Layer normalization works fine on RNNs and will work on very deep networks.
2. Having very small dimenstion of features may be a problem, since we normalize in that dimension. Yet I think it will still work and not cause much harm.
3. Regularization forces the network to kind of use all the features equally, I.E. not have too large values of W. This mean the range of values in a single layer will be closer to each other than with smaller regularization, and later normalization will not change those values too much.

```python

```


# Selección óptima de portafolios II

Entonces, tenemos que:
- La LAC describe las posibles selecciones de riesgo-rendimiento entre un activo libre de riesgo y un activo riesgoso.
- Su pendiente es igual al radio de Sharpe del activo riesgoso.
- La asignación óptima de capital para cualquier inversionista es el punto tangente de la curva de indiferencia del inversionista con la LAC.

Para todo lo anterior, supusimos que ya teníamos el portafolio óptimo (activo riesgoso).

En la clase pasada aprendimos a hallar este portafolio óptimo si el conjunto de activos riesgosos estaba conformado únicamente por dos activos:

$$w_{1,EMV}=\frac{(E[r_1]-r_f)\sigma_2^2-(E[r_2]-r_f)\sigma_{12}}{(E[r_2]-r_f)\sigma_1^2+(E[r_1]-r_f)\sigma_2^2-((E[r_1]-r_f)+(E[r_2]-r_f))\sigma_{12}}.$$

- Sin embargo, la complejidad del problema crece considerablemente con el número de variables, y la solución analítica deja de ser viable cuando mencionamos que un portafolio bien diversificado consta aproximadamente de 50-60 activos.
- En esos casos, este problema se soluciona con rutinas numéricas que hagan la optimización por nosotros, porque son una solución viable y escalable a más variables.

**Objetivos:**
- ¿Cuál es el portafolio óptimo de activos riesgosos cuando tenemos más de dos activos?
 - ¿Cómo construir la frontera de mínima varianza cuando tenemos más de dos activos?

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Maximizando el radio de Sharpe

### ¿Qué pasa si tenemos más de dos activos riesgosos?

En realidad es algo muy similar a lo que teníamos con dos activos.
- Para dos activos, construir la frontera de mínima varianza es trivial: todas las posibles combinaciones.
- Con más de dos activos, recordar la definición: la frontera de mínima varianza es el lugar geométrico de los portafolios que proveen el mínimo riesgo para un nivel de rendimiento dado.

 Ver en el tablero.

Analíticamente:
- $n$ activos,
- caracterizados por $(\sigma_i,E[r_i])$,
- cada uno con peso $w_i$, con $i=1,2,\dots,n$.

Entonces, buscamos los pesos tales que

\begin{align}
\min_{w_1,\dots,w_n} & \quad \sum_{i=1}^{n}w_i^2\sigma_i^2+\sum_{i=1}^{n}\sum_{j=1,j\neq i}^{n}w_iw_j\sigma_{ij}\\
\text{s.a.}          & \quad \sum_{i=1}^{n}w_i=1, w_i\geq0\\
                     & \quad \sum_{i=1}^{n}w_iE[r_i]=\bar{\mu},
\end{align}

donde $\bar{\mu}$ corresponde a un nivel de rendimiento objetivo.

**Obviamente, tendríamos que resolver este problema para muchos niveles de rendimiento objetivo.**

-  Explicar relación con gráfica.

-  Recordar clase 10.

Lo anterior se puede escribir vectorialmente como:

\begin{align}
\min_{\boldsymbol{w}} & \quad \boldsymbol{w}^T\Sigma\boldsymbol{w}\\
\text{s.a.}          & \quad \boldsymbol{1}^T\boldsymbol{w}=1, \boldsymbol{w}\geq0\\
                     & \quad E[\boldsymbol{r}^T]\boldsymbol{w}=\bar{\mu},
\end{align}

donde:
- $\boldsymbol{w}=\left[w_1,\dots,w_n\right]^T$ es el vector de pesos,
- $\boldsymbol{1}=\left[1,\dots,1\right]^T$ es un vector de unos,
- $E[\boldsymbol{r}]=\left[E[r_1],\dots,E[r_n]\right]^T$ es el vector de rendimientos esperados, y
- $\Sigma=\left[\begin{array}{cccc}\sigma_{1}^2 & \sigma_{12}  & \dots  & \sigma_{1n} \\
                                   \sigma_{21}  & \sigma_{2}^2 & \dots  & \sigma_{2n} \\
                                   \vdots       & \vdots       & \ddots & \vdots      \\
                                   \sigma_{n1}  & \sigma_{n2}  & \dots  & \sigma_{n}^2\end{array}\right]$ es la matriz de varianza-covarianza.

**Esta última forma es la que comúnmente usamos al programar, por ser eficiente y escalable a problemas de N variables.**

### Entonces, ¿para cuántos niveles de rendimiento objetivo tendríamos que resolver el anterior problema con el fin de graficar la frontera de mínima varianza?
- Observar que el problema puede volverse muy pesado a medida que incrementamos el número de activos en nuestro portafolio...
- Una tarea bastante compleja.

### Sucede que, en realidad, sólo necesitamos conocer dos portafolios que estén sobre la *frontera de mínima varianza*.
- Si logramos encontrar dos portafolios sobre la frontera, entonces podemos a la vez encontrar todas las posibles combinaciones de estos dos portafolios para trazar la frontera de mínima varianza.
- Ver el caso de dos activos.

### ¿Qué portafolios usar?
Hasta ahora, hemos estudiando profundamente como hallar dos portafolios muy importantes que de hecho yacen sobre la frontera de mínima varianza:
1. Portafolio de EMV: máximo SR.
2. Portafolio de mínima varianza: básicamente, el mismo problema anterior, sin la restricción de rendimiento objetivo.

Luego, tomar todas las posibles combinaciones de dichos portafolios usando las fórmulas para dos activos de medias y varianzas:
- w: peso para el portafolio EMV,
- 1-w: peso para le portafolio de mínima varianza.

## 2. Ejemplo ilustrativo.

Retomamos el ejemplo de mercados de acciones en los países integrantes del $G5$: EU, RU, Francia, Alemania y Japón.

```python
# Importamos pandas y numpy
import pandas as pd
import numpy as np
```

```python
# Resumen en base anual de rendimientos esperados y volatilidades
annual_ret_summ = pd.DataFrame(columns=['EU', 'RU', 'Francia', 'Alemania', 'Japon'], index=['Media', 'Volatilidad'])
annual_ret_summ.loc['Media'] = np.array([0.1355, 0.1589, 0.1519, 0.1435, 0.1497])
annual_ret_summ.loc['Volatilidad'] = np.array([0.1535, 0.2430, 0.2324, 0.2038, 0.2298])

annual_ret_summ.round(4)
```

      EU
      RU
      Francia
      Alemania
      Japon

      Media
      0.1355
      0.1589
      0.1519
      0.1435
      0.1497

      Volatilidad
      0.1535
      0.243
      0.2324
      0.2038
      0.2298

```python
# Matriz de correlación
corr = pd.DataFrame(data= np.array([[1.0000, 0.5003, 0.4398, 0.3681, 0.2663],
                                    [0.5003, 1.0000, 0.5420, 0.4265, 0.3581],
                                    [0.4398, 0.5420, 1.0000, 0.6032, 0.3923],
                                    [0.3681, 0.4265, 0.6032, 1.0000, 0.3663],
                                    [0.2663, 0.3581, 0.3923, 0.3663, 1.0000]]),
                    columns=annual_ret_summ.columns, index=annual_ret_summ.columns)
corr.round(4)
```

      EU
      RU
      Francia
      Alemania
      Japon

      EU
      1.0000
      0.5003
      0.4398
      0.3681
      0.2663

      RU
      0.5003
      1.0000
      0.5420
      0.4265
      0.3581

      Francia
      0.4398
      0.5420
      1.0000
      0.6032
      0.3923

      Alemania
      0.3681
      0.4265
      0.6032
      1.0000
      0.3663

      Japon
      0.2663
      0.3581
      0.3923
      0.3663
      1.0000

```python
# Tasa libre de riesgo
rf = 0.05
```

Esta vez, supondremos que tenemos disponibles todos los mercados de acciones y el activo libre de riesgo.

#### 1. Construir la frontera de mínima varianza

##### 1.1. Encontrar portafolio de mínima varianza

```python
# Importamos funcion minimize del modulo optimize de scipy
from scipy.optimize import minimize
```

```python
np.diag(np.array([1, 2, 3, 4]))
```

    array([[1, 0, 0, 0],
           [0, 2, 0, 0],
           [0, 0, 3, 0],
           [0, 0, 0, 4]])

```python

## Construcción de parámetros
# 1. Sigma: matriz de varianza-covarianza Sigma = S.dot(corr).dot(S)
S = np.diag(annual_ret_summ.loc['Volatilidad'].values)
Sigma = S.dot(corr).dot(S)
# 2. Eind: rendimientos esperados activos individuales
Eind = annual_ret_summ.loc['Media'].values

```

```python
# Función objetivo
def var(w, Sigma):
    return w.T.dot(Sigma).dot(w)
```

```python
# Número de activos
N = len(Eind)
# Dato inicial
w0 = np.ones(N)/N
# Cotas de las variables
bnds = ((0, 1), ) * N
# Restricciones
cons = {'type': 'eq', 'fun': lambda w: w.sum() - 1}
```

```python
# Portafolio de mínima varianza
minvar = minimize(fun=var,
                  x0=w0,
                  args=(Sigma,),
                  bounds=bnds,
                  constraints=cons)
minvar
```

         fun: 0.01861776391061502
         jac: array([0.03718246, 0.03881475, 0.03859101, 0.03755156, 0.0370423 ])
     message: 'Optimization terminated successfully'
        nfev: 42
         nit: 7
        njev: 7
      status: 0
     success: True
           x: array([6.17797049e-01, 3.46944695e-18, 0.00000000e+00, 2.09394358e-01,
           1.72808594e-01])

```python
# Pesos, rendimiento y riesgo del portafolio de mínima varianza
w_minvar = minvar.x
E_minvar = Eind.T.dot(w_minvar)
s_minvar = var(w_minvar, Sigma)**0.5
RS_minvar = (E_minvar - rf) / s_minvar
w_minvar, E_minvar, s_minvar, RS_minvar
```

    (array([6.17797049e-01, 3.46944695e-18, 0.00000000e+00, 2.09394358e-01,
            1.72808594e-01]),
     0.13962903688859557,
     0.13644692708381168,
     0.6568783834431207)

##### 1.2. Encontrar portafolio EMV

```python
# Función objetivo
def menos_RS(w, Eind, rf, Sigma):
    E_port = Eind.T.dot(w)
    s_port = var(w, Sigma)**0.5
    RS = (E_port - rf) / s_port
    return - RS
```

```python
# Número de activos
N = len(Eind)
# Dato inicial
w0 = np.ones(N)/N
# Cotas de las variables
bnds = ((0, 1), ) * N
# Restricciones
cons = {'type': 'eq', 'fun': lambda w: w.sum() - 1}
```

```python
# Portafolio EMV
emv = minimize(fun=menos_RS,
                  x0=w0,
                  args=(Eind, rf, Sigma),
                  bounds=bnds,
                  constraints=cons)
emv
```

         fun: -0.6644372965632436
         jac: array([-0.3608895 , -0.36076408, -0.36036385, -0.36108153, -0.36062376])
     message: 'Optimization terminated successfully'
        nfev: 30
         nit: 5
        njev: 5
      status: 0
     success: True
           x: array([0.50714174, 0.07470888, 0.02471533, 0.18943972, 0.20399434])

```python
# Pesos, rendimiento y riesgo del portafolio EMV
w_emv = emv.x
E_emv = Eind.T.dot(w_emv)
s_emv = var(w_emv, Sigma)**0.5
RS_emv = (E_emv - rf) / s_emv
w_emv, E_emv, s_emv, RS_emv
```

    (array([0.50714174, 0.07470888, 0.02471533, 0.18943972, 0.20399434]),
     0.1420657564717856,
     0.13856199365687238,
     0.6644372965632436)

```python
w_minvar, E_minvar, s_minvar, RS_minvar
```

    (array([6.17797049e-01, 3.46944695e-18, 0.00000000e+00, 2.09394358e-01,
            1.72808594e-01]),
     0.13962903688859557,
     0.13644692708381168,
     0.6568783834431207)

```python
annual_ret_summ.columns
```

    Index(['EU', 'RU', 'Francia', 'Alemania', 'Japon'], dtype='object')

##### 1.3. Construir frontera de mínima varianza

También debemos encontrar la covarianza (o correlación) entre estos dos portafolios:

```python
# Covarianza entre los portafolios
cov_emv_minvar = w_emv.T.dot(Sigma).dot(w_minvar)
cov_emv_minvar
```

    0.018690275386034134

```python
# Correlación entre los portafolios
corr_emv_minvar = cov_emv_minvar / (s_emv * s_minvar)
corr_emv_minvar
```

    0.9885708894197612

```python
# Vector de w
w_p = np.linspace(0, 5)
```

```python
# DataFrame de portafolios:
# 1. Índice: i
# 2. Columnas 1-2: w, 1-w
# 3. Columnas 3-4: E[r], sigma
# 4. Columna 5: Sharpe ratio
frontera = pd.DataFrame(data={'Media': w_p * E_emv + (1 - w_p) * E_minvar,
                              'Vol': ((w_p * s_emv)**2 + ((1 - w_p) * s_minvar)**2 + 2 * w_p * (1 - w_p) * cov_emv_minvar)**0.5})
frontera['RS'] = (frontera['Media'] - rf) /frontera['Vol']
frontera.head()
```

      Media
      Vol
      RS

      0
      0.139629
      0.136447
      0.656878

      1
      0.139878
      0.136518
      0.658359

      2
      0.140126
      0.136622
      0.659677

      3
      0.140375
      0.136759
      0.660833

      4
      0.140624
      0.136930
      0.661827

```python
# Importar librerías de gráficos
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a SR, los activos individuales
# y los portafolios hallados
plt.figure(figsize=(10, 6))
# Frontera
plt.scatter(frontera['Vol'], frontera['Media'], c = frontera['RS'], cmap='RdYlBu', label = 'Frontera de minima varianza')
plt.colorbar()
# Activos ind
for activo in annual_ret_summ.columns:
    plt.plot(annual_ret_summ.loc['Volatilidad', activo],
             annual_ret_summ.loc['Media', activo],
             'o',
             ms=5,
             label = activo)
# Port. óptimos
plt.plot(s_minvar, E_minvar, '*g', ms=10, label='Portafolio de mínima varianza')
plt.plot(s_emv, E_emv, '*r', ms=10, label='Portafolio eficiente en media varianza')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
plt.grid()
plt.legend(loc='best')
```

**A partir de lo anterior, solo restaría construir la LAC y elegir la distribución de capital de acuerdo a las preferencias (aversión al riesgo).**
___

## 3. Comentarios finales

### 3.1. Restricciones adicionales

Los inversionistas pueden tener restricciones adicionales:
1. Restricciones en posiciones cortas.
2. Pueden requerir un rendimiento mínimo.
3. Inversión socialmente responsable: prescinden de inversiones en negocios o paises considerados éticamente o políticamente indeseables.

Todo lo anterior se puede incluir como restricciones en el problema de optimización, y puede ser llevado a cabo a costa de un cociente de Sharpe menor.

### 3.2. Críticas a la optimización media varianza
1. Solo importan medias y varianzas: recordar que la varianza subestima el riesgo en algunos casos.
2. Preferencias media-varianza tratan las ganancias y pérdidas simétricamente: el sentimiento de insatisfacción de una perdida es mayor al sentimiento de satisfacción de una ganancia (aversión a pérdidas).
3. La aversión al riesgo es constante: la actitud frente al riesgo puede cambiar, por ejemplo con el estado de la economía.
4. Horizonte corto (un periodo).
5. Basura entra - basura sale: la optimización media varianza es supremamente sensible a las entradas: estimaciones de rendimientos esperados y varianzas.
___

# Anuncios parroquiales

## 1. Quiz la próxima clase (clases 12, 13, y 14).
## 2. Revisar archivo Tarea 6. Viernes, 23 de Octubre.
## 3. [Nota interesante](http://yetanothermathprogrammingconsultant.blogspot.com/2016/08/portfolio-optimization-maximize-sharpe.html)
## 4. Martes 3 y Viernes 6 de Noviembre, ¡NO HAY CLASE!
## 5. Clase de repaso módulo 3 - Jueves 22 de Octubre.
## 7. Clases de repaso módulos 3 y 4 - Jueves 12 de Noviembre.
## 8. Examen final: Viernes 13 de Noviembre.

Created with Jupyter by Esteban Jiménez Rodríguez.


# CSE 555 - Introduction to pattern recognition
## Problem Set 1: Linear Discriminant Functions and Support Vector Machines

## PART 1

Dataloader to read the MNIST dataset. The dataset is read and is flattened. Hence each 28x28 image is now a vector of length 784.

```python
#Using data loader from previous assignment

import numpy as np

def load_mnist(kind='train'):
    import os
    import gzip

    """Load MNIST data from `path`"""
    labels_path = ('%s-labels-idx1-ubyte.gz' % kind)
    images_path = ('%s-images-idx3-ubyte.gz' % kind)

    with gzip.open(labels_path, 'rb') as lbpath:
        labels = np.frombuffer(lbpath.read(), dtype=np.uint8,
                               offset=8)

    with gzip.open(images_path, 'rb') as imgpath:
        images = np.frombuffer(imgpath.read(), dtype=np.uint8,
                               offset=16).reshape(len(labels), 784)

    return images, labels

x_train, y_train = load_mnist(kind='train')
x_test, y_test = load_mnist(kind='t10k')
```

Preprocessing the data so that the mean of the dataset is zero and the standard deviation is one.

```python
from sklearn import preprocessing
scaler = preprocessing.StandardScaler().fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)
```

The penalty is chosen as 'l1' since the requirement is to use '1-norm soft margin'.
The regularization parameter (C) is a hyper-parameter. This is tuned using cross-validation.

```python
from sklearn import svm
from sklearn.model_selection import GridSearchCV

parameters = {'C':[0.001, 0.01, 0.1, 1]}
svc = svm.LinearSVC(loss='squared_hinge', penalty='l1', dual=False)
clf = GridSearchCV(svc, parameters)
# clf = svm.LinearSVC(penalty='l1')
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
```

The best value for regularization parameter is obtained through cross-validation.

```python
print("Best C from cross validation: ", clf.best_params_)
```

    Best C from cross validation:  {'C': 0.1}

The performance of the model is evaluated through accuracy and confusion matrix as metrics.

```python
from sklearn.metrics import confusion_matrix
print("score = %3.2f" %(clf.score(x_test, y_test)))
print("confusion matrix: \n ", confusion_matrix(y_test, y_pred))
```

## PART 2

Given:

features
\begin{equation}
\left( x_{1} ,\ y_{1}\right) ,...,\left( x_{N} ,\ y_{N}\right) \ where\ y_{1} ,...,y_{N} \ \in \{-1,\ 1\}
\end{equation}

We need to minimize:
\begin{equation*}
W^{T} .W+C\sum ^{N}_{i=1} \xi _{i}
\end{equation*}

i.e., the weighted sum between the squared length of the separating vector and the errors, where $\displaystyle W$ is the separating vector.

s.t
\begin{equation*}
y_{i} .\left( W^{T} .x_{i}\right) \geq 1-\xi _{i}
\end{equation*}

and
\begin{gather*}
\xi _{i} \geq 0
\end{gather*}

for $\displaystyle i=1,...,N$.

### Objective: To form the Lagrangian dual problem

Using slide 4/27:
The primal margin is :

\begin{gather*}
\gamma \ =\ \frac{1}{\sqrt{W^{T} .W+C\sum ^{N}_{i=1} \xi _{i}}}\\
\end{gather*}

Using slide 5/27:
Lagrange function:

\begin{equation*}
L=\overrightarrow{W}^{T}\overrightarrow{W} +C\sum ^{N}_{i=1} \xi _{i} +\sum ^{N}_{i=1} α_{i}\left( 1−y_{i} \cdot W^{T} x_{i} -\xi _{i}\right) -\sum ^{N}_{i=1} \beta _{i} \xi _{i}
\end{equation*}
with

Lagrange multipliers: $\displaystyle \alpha _{i} \geq 0\ ,\ \beta _{i} \geq 0$,

Inequality constraits: $\displaystyle 1−𝑦_{i}\left( W^{T} x_{i}\right) −\xi _{i} \leq 0$ and $\displaystyle \xi _{i} \geq 0$

for all $\displaystyle i=1,...,N$
#### Claim:

$\underbrace{\underset{\alpha,\beta}{max} \underset{W,\xi}{min}{L}}_\text{dual solution} \leq \underbrace{\underset{W,\xi}{min} \underset{\alpha,\beta}{max}{L}}_\text{primal solution}$

\begin{equation*}
L=\overrightarrow{W}^{T}\overrightarrow{W} +C\sum ^{N}_{i=1} \xi _{i} +\sum ^{N}_{i=1} α_{i}\left( 1−y_{i} \cdot W^{T} x_{i} -\xi _{i}\right) -\sum ^{N}_{i=1} \beta _{i} \xi _{i}
\end{equation*}

\begin{equation*}
\\
  {\partial_{{W}}}{L} = 2W-\sum ^{N}_{i=1} \alpha_i y_i \overrightarrow{x_i} \stackrel{\text{set}}{=} 0 \implies 2W = {\sum ^{N}_{i=1} \alpha_i y_i \overrightarrow{x_i}}
  \\
  {\partial_{{\xi}}}{L} = C - \alpha_i - \beta_i = 0
  \\
  \implies 0 \leq \alpha_i \leq C \text{ and } \beta_i \geq 0
\end{equation*}

Second order partial derivative \begin{gather*} \partial^2_w\mathcal L=1 \gt 0:
\end{gather*}

\begin{gather*}
2W=\sum ^{N}_{i=1}\alpha_i y_i \vec x_i\ \text{minimizes L with given } \alpha_i,\forall i \end{gather*}

Substituting \begin{gather*} 2W = {\sum ^{N}_{i=1} \alpha_i y_i \overrightarrow{x_i}}, C=\alpha_i + \beta_i \end{gather*}
into Lagrange function, we get the dual problem of maximizing:

\begin{equation*}
{{L}} = \overrightarrow{W}^{T} \overrightarrow{W} + (\alpha_i + \beta_i) \sum_{i=1}^N\xi_i + \sum_{i=1}^N\alpha_i(1-y_i.\overrightarrow{W}^{T} \overrightarrow{x_i} -\xi_i) - \sum_{i=1}^N \beta_i \xi_i
\\\quad = W^T \overrightarrow{W} + \alpha_i\sum_{i=1}^N\xi_i + \beta_i \sum_{i=1}^N\xi_i +\sum_{i=1}^N \alpha_i - \overrightarrow{W}^{T} \sum_{i=1}^N \alpha_i y_i \overrightarrow{x_i} - \alpha_i \sum_{i=1}^N \xi_i - \beta_i \sum_{i=1}^N \xi_i
\\\quad = W^T \overrightarrow{W} + \alpha_i\sum_{i=1}^N\xi_i + \beta_i \sum_{i=1}^N\xi_i +\sum_{i=1}^N \alpha_i - \overrightarrow{W}^{T} 2\overrightarrow{W} - \alpha_i \sum_{i=1}^N \xi_i - \beta_i \sum_{i=1}^N \xi_i
\end{equation*}

\begin{equation*}
\\=\sum_{i=1}^N \alpha_i-\sum_{i,j=1}^N \overrightarrow{W}^{T}\alpha_i y_i \overrightarrow{x_i}
\end{equation*}
\begin{equation*}
\\=\sum_{i=1}^N \alpha_i-\sum_{i,j=1}^N \alpha_i \alpha_j y_i y_j \overrightarrow{x_i}^{T}\overrightarrow{x_j}
\end{equation*}

**The dual margin is :**

\begin{gather*}
\gamma \ =\ \frac{1}{\sqrt{\alpha_i \alpha_j y_i y_j \overrightarrow{x_i}^{T}\overrightarrow{x_j}}}\\
\end{gather*}

#### Benefits of maximizing the margin
1. Increasing the margin leads to ignoring the points close to the boundary. That is, the samples inside the margins are penalized less.
2. Having a high margin in cases of data with outliers would be better so that the algorithm can generalize.
3. Having a high margin would help to avoid overfitting.

#### Support vectors characteristics:
1. Support vectors are the points that are crucial in building SVM.
2. These are data points that are closer to the hyperplane and influence the position and orientation of the hyperplane.
3. The support vectors are those points for which the Lagrange multiplier is not zero.

#### Benefits of solving the dual problem instead of the primal problem
1. The biggest advantage of solving the dual problem comes when the kernel trick is used. It allows us to classify data that is not linearly separable in the original feature space.
2. In dual problem, it is not necessary to explicitly compute the mapping for each data point as in the case of the primal problem.
3. The dual space has only as many parameters as the number of data points irrespective of the dimensions.
4. Regularizing the sparse support vector associated with the dual hypothesis is sometimes more intuitive than regularizing the vector of regression coefficients.