# Le-Net 1 based architecture

We start with 41X41 (I) after first convolution (9x9)we have 33X33 (L1). The next pooling layer reduces dimension with 3 to an output image of 11X11 with 4x4 pooling kernels (L2). Then we apply different types of convolution 4x4 kernels on the L2 layer resulting in 8x8 (L3) . Then followed by pooling 2X2 resulting in 4x4 output map (L4). So we have 16 connection for each element in layer L4 (which depend on the amount of different Covolutions in L3)

\begin{equation}
f(x)=\frac{1}{1+e^{-x}} \\
F_{k}= f( \sum_{i} \mathbf{W^{k}_{i} \cdot y_{i}}-b_{k})
\end{equation}

\begin{equation}
E=\sum_{k} \frac{1}{2}|t_k-F_{k}|^{2} \\
\Delta W_{ij}= - \eta \frac{dE}{d W_{ij}}
\end{equation}

\begin{equation}
\Delta W_{ij}= \sum_{k} - \eta \frac{dE}{d F_{k}} \frac{dF_{k}}{dx_{k}} \frac{dx_{k}}{dW_{ij}}=\sum_{k} \eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} \frac{dx_{k}}{dW_{ij}} \\
= \eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} y_{ij}
\end{equation}

\begin{equation}
\Delta b_{k}= - \eta \frac{dE}{d F_{k}} \frac{dF_{k}}{dx_{k}} \frac{dx_{k}}{b_{k}}=\eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} \cdot-1
\end{equation}

Since $\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}}$ is always positive we can neglect this term in our programme

\begin{equation}
x_{k}=\sum_{ij} W^{k}[i,j] \; y^{4rb}[i,j] - b_{k}
\end{equation}

\begin{equation}
y^{4rb}[i,j]= \sum_{u,v} W^{3rb}[u,v] \; y^{3rb} [2i+u,2j+v]
\end{equation}

\begin{equation}
y^{3rb} [2i+u,2j+v]= f\left (x^{3rb}[2i+u,2j+v] \right)
\end{equation}

\begin{equation}
x^{3rb}[2i+u,2j+v]=\sum_{nm} W^{2rb}[n,m] \; y^{2rb}[n+(2i+u),m+(2j+v)] -b^{3rb}[2i+u,2j+v]
\end{equation}

\begin{equation}
\begin{split}
\Delta W^{2rb}[n,m] =\sum_{k} - \eta  \frac{dE}{dF_{k}}
\frac{dF_{k}}{dx_{k}}
\sum_{ij} \frac{dx_{k}}{dy^{4rb}[i,j]}
 \sum_{uv}\frac{dy^{4rb}[i,j]}{d y^{3rb} [2i+u,2j+v]}
\frac{d y^{3rb} [2i+u,2j+v]}{d x^{3rb}[2i+u,2j+v]}
\sum_{nm}\frac{d x^{3rb}[2i+u,2j+v]}{d W^{2rb}[n,m]}
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\Delta b^{3rb}[2i+u,2j+v] =\sum_{k} - \eta  \frac{dE}{dF_{k}}
\frac{dF_{k}}{dx_{k}}
\sum_{ij} \frac{dx_{k}}{dy^{4rb}[i,j]}
 \sum_{uv}\frac{dy^{4rb}[i,j]}{d y^{3rb} [2i+u,2j+v]}
\frac{d y^{3rb} [2i+u,2j+v]}{d x^{3rb}[2i+u,2j+v]}
\frac{d x^{3rb}[2i+u,2j+v]}{d b^{3rb}[2i+u,2j+v]}
\end{split}
\end{equation}

\begin{equation}
    \frac{dx_{k}}{dy^{4rb}[i,j]} = W^{4rbk}[i,j]\\
\end{equation}

\begin{equation}
    \frac{dy^{4rb}[i,j]}{d y^{3rb} [2i+u,2j+v]} = W^{3rb}[u,v] \\
 \end{equation}

\begin{equation}
\frac{d y^{3rb} [2i+u,2j+v]}{d x^{3rb}[2i+u,2j+v]}=\frac{e^{-x^{3rb}[2i+u,2j+v]}}{(1+e^{-x^{3rb}[2i+u,2j+v]})^2}
\end{equation}
This term is first not included since it is always positive. If the training will not converge it might be possible to include this term

 \begin{equation}
    \frac{d y^{3rb} [2i+u,2j+v]}{d W^{2rb}[n,m]}= y^{2rb} [n+(2i+u),m+(2j+v)] \\
\end{equation}

\begin{equation}
\frac{d x^{3rb}[2i+u,2j+v]}{d b^{3rb}[2i+u,2j+v]}=-1
\end{equation}

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from numpy import linalg as lin
import scipy.signal as sig
from PIL import Image
import glob
import matplotlib.cm as cm
import itertools
```

```python
########### Load Input ############################################################################################################################
# In this script I used the brightness to determine structures, instead of one RGB color:
# this is determined by: 0.2126*R + 0.7152*G + 0.0722*B
# Source: https://en.wikipedia.org/wiki/Relative_luminance

patchSize=41 # patchsize this must be 48 since our network can only handle this value

# Open forest
Amount_data= len(glob.glob('Forest/F*'))
dataPatchedF=[]
dataF_RGB=[]

for k in range (0, Amount_data):
    name="Forest/F%d.png" % (k+1)
    img = Image.open(name)
    data=img.convert('RGB')
    data= np.asarray( data, dtype="int32" )
    data=0.2126*data[:,:,0]+0.7152*data[:,:,1]+0.0722*data[:,:,2]
    data2=img.convert('RGB')
    data2= np.asarray( data2, dtype="int32" )

    Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
    Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
    dataPatchedF.append(np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize] for i in range(0,Xamount)] for j in range(0,Yamount)]))
    for m in range(0,3):
        dataF_RGB.append(np.array([[data2[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,m] for i in range(0,Xamount)] for j in range(0,Yamount)]))

# Open city
Amount_data= len(glob.glob('City/C*'))
dataPatchedC=[]
dataC_RGB=[]

for k in range (0, Amount_data):
    name="City/C%d.png" % (k+1)
    img = Image.open(name)
    data=img.convert('RGB')
    data = np.asarray( data, dtype="int32" )
    data=0.2126*data[:,:,0]+0.7152*data[:,:,1]+0.0722*data[:,:,2]
    data2=img.convert('RGB')
    data2= np.asarray( data2, dtype="int32" )

    Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
    Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
    dataPatchedC.append(np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize] for i in range(0,Xamount)] for j in range(0,Yamount)]))
    for m in range(0,3):
        dataC_RGB.append(np.array([[data2[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,m] for i in range(0,Xamount)] for j in range(0,Yamount)]))

# Open water
dataPatchedW=[]
Amount_data= len(glob.glob('Water/W*'))
dataW_RGB=[]

for k in range (0, Amount_data):
    name="Water/W%d.png" % (k+1)
    img = Image.open(name)
    data=img.convert('RGB')
    data = np.asarray( data, dtype="int32" )
    data=0.2126*data[:,:,0]+0.7152*data[:,:,1]+0.0722*data[:,:,2]
    Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
    Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
    dataPatchedW.append(np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize] for i in range(0,Xamount)] for j in range(0,Yamount)]))
    #for m in range(0,3):
        # dataW_t=np.array([[data2[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,m] for i in range(0,Xamount)] for j in range(0,Yamount)]))

```

```python
A=np.array([[data2[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,m] for i in range(0,Xamount)] for j in range(0,Yamount)])
```

```python
A.shape
```

    (13, 32, 41, 41)

```python
########### Functions ############################################################################################################################

# Define Activitation functions, pooling and convolution functions (the rules)

def Sigmoid(x):
    return (1/(1+np.exp(-x)))

def Sigmoid_dx(x):
    return np.exp(-x)/((1+np.exp(-x))**2)

def TanH(x):
    return (1-np.exp(-x))/(1+np.exp(-x))

def Pool(I,W):
    PoolImg=np.zeros((len(I)/len(W),len(I)/len(W))) # W must fit an integer times into I.
    for i in range(0,len(PoolImg)):
        for j in range(0,len(PoolImg)):
            SelAr=I[i*len(W):(i+1)*len(W),j*len(W):(j+1)*len(W)]
            PoolImg[i,j]=np.inner(SelAr.flatten(),W.flatten()) # Now this is just an inner product since we have vectors
    return PoolImg

# To automatically make Gaussian kernels
def makeGaussian(size, fwhm = 3, center=None):
    x = np.arange(0, size, 1, float)
    y = x[:,np.newaxis]

    if center is None:
        x0 = y0 = size // 2
    else:
        x0 = center[0]
        y0 = center[1]

    return np.exp(-4*np.log(2) * ((x-x0)**2 + (y-y0)**2) / fwhm**2)

# To automatically define pooling nodes
def Pool_node(N):
    s=(N,N)
    a=float(N)*float(N)
    return (1.0/a)*np.ones(s)

```

```python
#################### Define pooling layers ###########################################################################
P12=Pool_node(3)*(1.0/100.0) #factor 1000 added to lower values more
P34=Pool_node(2)*(1.0/100.0)

#################### Define Convolution layers #######################################################################

######### First C layer #########
C1=[]

## First Kernel

# Inspiration: http://en.wikipedia.org/wiki/Sobel_operator
# http://stackoverflow.com/questions/9567882/sobel-filter-kernel-of-large-size

Kernel=np.array([[4,3,2,1,0,-1,-2,-3,-4],
                 [5,4,3,2,0,-2,-3,-4,-5],
                 [6,5,4,3,0,-3,-4,-5,-6],
                 [7,6,5,4,0,-4,-5,-6,-7],
                 [8,7,6,5,0,-5,-6,-7,-8],
                 [7,6,5,4,0,-4,-5,-6,-7],
                 [6,5,4,3,0,-3,-4,-5,-6],
                 [5,4,3,2,0,-2,-3,-4,-5],
                 [4,3,2,1,0,-1,-2,-3,-4]])

C1.append(Kernel)

## Second Kernel
Kernel=np.matrix.transpose(Kernel)
C1.append(Kernel)

##Third Kernel
Kernel=makeGaussian(9,5)
Kernel=(1/np.sum(Kernel))*Kernel
C1.append(Kernel)

######### Initialize output weights and biases #########

# Define the number of branches in one row
N_branches= 3
ClassAmount=3 # Forest, City, Water
Size_C2=4
S_H3=((patchSize-C1[0].shape[0]+1)/P12.shape[1])-Size_C2+1
S_H4=S_H3/P34.shape[1]

C2=np.random.rand(len(C1),N_branches, 4, 4) # second convolution weigths
W=np.random.rand(ClassAmount, len(C1), N_branches, S_H4, S_H4) # end-weight from output to classifier-neurons
H3_bias=np.random.rand(len(C1),N_branches) #bias in activation function from C2 to H3
Output_bias=np.random.rand(ClassAmount)# bias on the three classes

#learning rates
n_bias=1*10**-2
n_W=1*10**-2
n_C2=1*10**-2
n_H3_bias=1*10**-2
#Labda=5*10**-3

```

```python
N_plts=len(C1)
for i in range(0,N_plts):
    plt.subplot(4,3,i+1)
    plt.imshow(C1[i])
```

# For the extra information regarding the code in the following cell

a random patch is chosen in the following way: the program counts how many files and patches there are in total, then it permutes the sequence so that a random patch is chosen every iteration (forest, city, water). After selecting the number the file has to be found back.

```python
# Determines total amount of patches and makes it easier to find back the patches
N_F=len(dataPatchedF)
Patches_F=[]
j=0;

#Determines the total amount of patches N_F in forest
for i in range(0,N_F):
    N_ff=dataPatchedF[i].shape[0]*dataPatchedF[i].shape[1]
    Patches_F.append(np.reshape(dataPatchedF[i], (N_ff, patchSize, patchSize)))
    j=j+N_ff
N_F=j

#Determines the total amount of patches N_C in city
N_C=len(dataPatchedC)
Patches_C=[]
j=0;
for i in range(0,N_C):
    N_cc=dataPatchedC[i].shape[0]*dataPatchedC[i].shape[1]
    j=j+N_cc
    Patches_C.append(np.reshape(dataPatchedC[i], (N_cc, patchSize, patchSize)))
N_C=j

#Determines the total amount of patches N_W in water
N_W=len(dataPatchedW)
Patches_W=[]
j=0;
for i in range(0,N_W):
    N_ww=dataPatchedW[i].shape[0]*dataPatchedW[i].shape[1]
    Patches_W.append(np.reshape(dataPatchedW[i], (N_ww, patchSize, patchSize)))
    j=j+N_ww
N_W=j

N_total=N_F+N_C+N_W
Sequence = np.arange(N_total)
Sequence = np.random.permutation(Sequence)

# Make patches tractable
Patches_F = list(itertools.chain.from_iterable(Patches_F))
Patches_F=np.asarray(Patches_F)

Patches_C = list(itertools.chain.from_iterable(Patches_C))
Patches_C=np.asarray(Patches_C)

Patches_W = list(itertools.chain.from_iterable(Patches_W))
Patches_W=np.asarray(Patches_W)

```

```python
N_F
```

    5382

```python
len(Sequence)
```

    16120

```python
# TRAINING PHASE
delta_H4=np.zeros((len(C1), N_branches, S_H4, S_H4))
delta_H3=np.zeros((len(C1), N_branches, S_H4, S_H4))
from itertools import product
###### Chooses patch and defines label #####
#for PP in range(0,len(Sequence)):
for PP in range(0,1):
    SS=Sequence[PP]
    if SS<N_F:
        Class_label=np.array([1,0,0])
        inputPatch=Patches_F[SS]
    elif(SS>=N_F) and (SS<(N_F+N_C)):
        Class_label=np.array([0,1,0])
        inputPatch=Patches_C[SS-N_F]
    else:
        Class_label=np.array([0,0,1])
        inputPatch=Patches_W[SS-N_F-N_C]
    ### Layer 1 ###
    H1=[]
    H2=[]
    H3=np.zeros((len(C1), N_branches, S_H3,S_H3))
    H4=np.zeros((len(C1), N_branches, S_H4,S_H4))
    x=np.zeros(ClassAmount)
    f=np.zeros(ClassAmount)

    II=1

    for r in range (0, len(C1)):
        H1.append(sig.convolve(inputPatch, C1[r], 'valid'))
        H2.append(Pool(H1[r], P12))
        #From here on BP trakes place!
        ITER=0
        while II==1:
            for b in range(0,N_branches):
                H3[r][b]=Sigmoid(sig.convolve(H2[r], C2[r][b],'valid')-H3_bias[r][b])
                H4[r][b]=Pool(H3[r][b],P34)
    #Now we have 3x3x4x4 inputs, connected to the 3 output nodes
            for k in range(0,ClassAmount):
                x[k]=np.inner(H4.flatten(),W[k].flatten())
                f[k]=Sigmoid(x[k]-Output_bias[k])
            f=f/np.sum((f))

            ###### Back-propagation #####
            # First learning the delta's
            e_k=f-Class_label
            delta_k=e_k*Sigmoid_dx(x)
            Output_bias=Output_bias[k]+n_bias*e_k
            for k in range(0, ClassAmount):
                #update weights output layer
                W[k]=W[k]-n_W*delta_k[k]*H4
                #determine delta in Sub-sampling layer that is not trained
                delta_H4=delta_H4+delta_k[k]*W[k]
                #for b in range(0,N_branches):
                    #for r in range(0,len(C1):
                delta_H3=
            ERROR=np.sum((Class_label-f)**2)
            ITER=ITER+1
            if ERROR<0.55 or ITER>2:
                II=0

n_bias=1*10**-2
n_W=1*10**-2
n_C2=1*10**-2
n_H3_bias=1*10**-2

```

```python
N_F
```

    5382

```python
delta_k
```

```python
Class_label
```

```python
####### Test phase #######
Error_Test=[]
N_correct=0
for PP in range(9000, 10000):
    SS=Sequence[PP]
    if SS<N_F:
        Class_label=np.array([1,0,0])
        inputPatch=Patches_F[SS]
    elif(SS>=N_F) and (SS<(N_F+N_C)):
        Class_label=np.array([0,1,0])
        inputPatch=Patches_C[SS-N_F]
    else:
        Class_label=np.array([0,0,1])
        inputPatch=Patches_W[SS-N_F-N_C]
    ### Layer 1 ###
    H1=[]
    H2=[]
    H3=np.zeros((len(C1), N_branches, S_H3,S_H3))
    H4=np.zeros((len(C1), N_branches, S_H4,S_H4))
    x=np.zeros(ClassAmount)
    f=np.zeros(ClassAmount)
    for r in range (0, len(C1)):
        H1.append(sig.convolve(inputPatch, C1[r], 'valid'))
        H2.append(Pool(H1[r], P12))
        #From here on BP trakes place!
        for b in range(0,N_branches):
            H3[r][b]=Sigmoid(sig.convolve(H2[r], C2[r][b],'valid')-H3_bias[r][b])
            H4[r][b]=Pool(H3[r][b],P34)
    #Now we have 3x3x4x4 inputs, connected to the 3 output nodes
    for k in range(0,ClassAmount):
        x[k]=np.inner(H4.flatten(),W[k].flatten())
        f[k]=Sigmoid(x[k]-Output_bias[k])
    f=f/np.sum((f))
    Error_Test.append(np.sum((Class_label-f)**2))
            ###### Back-propagation #####
            # First learning the delta's

    if np.argmax(f)==np.argmax(Class_label):
        print True
        N_correct=N_correct+1
    else:
        print False

Perc_corr=float(N_correct)/1000
print Perc_corr
```

```python
float(N_correct)/1000
```

    0.324

```python
    ###### Back-propagation #####
    for k in range(0, ClassAmount):
        #bias[k]=bias[k]-n_bias*(Class_label[k]-f[k])
        for r in range (0, len(C1)):
            for b in range(0,N_branches):
                # W[r][b,k]=W[r][b,k]+n_W*(Class_label[k]-f[k])*H4[r][b]
                for i, j, u, v, n, m in product(range(0,4), range(0,4), range(0,2), range(0,2), range(0,4), range(0,4)):
                    C2[r][b,n,m]=C2[r][b,n,m]+n_C2*(Class_label[k]-f[k])*W[r][b,k,i,j]*H2[r][n+(2*j+u),m+(2*j+v)]
                    H3_bias[r][b,2*i+u,2*j+v]=H3_bias[r][b,2*i+u,2*j+v]-n_H3_bias*(Class_label[k]-f[k])*W[r][b,k,i,j]

    print(Class_label,f)
    if (PP%100==0):print(PP)
```

# save training parameters

```python
with open("C2.txt", 'w') as f:
    f.write(str(C2))
with open("W.txt", 'w') as f:
    f.write(str(W))
with open("H3_bias.txt", 'w') as f:
    f.write(str(H3_bias))
with open("Bias.txt", 'w') as f:
    f.write(str(bias))

```

```python
print Class_label
```

    [0 1 0]

```python

```


```python
from IPython.display import HTML
HTML('''
The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click <a href="javascript:code_toggle()">here</a>.''')
```

# try solving the following pde:

$\partial_t y = \gamma \partial_x(x^{-2}y) + \beta\partial^2_x(x^{-1}y)$,

with initial condition
$y(x,t=0) = const \in [1-\delta x,1+\delta x]$

```python
%matplotlib inline
#%pylab inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from sympy.solvers.pde import pdsolve
from sympy import Function,diff,Eq
from sympy.abc import x,y,a,b

f=Function('f')
u= f(x,y)
ux = u.diff(x)
uy = u.diff(y)
uyy = u.diff(y,y)
gam = 0.1
beta = 0. #0.25*gam*0.05
print gam,beta
b=0
#eq = Eq(ux-2*(beta-gam)*u/y**3 + (gam-2*beta)*uy/y**2 - beta*uyy/y)
eq = Eq(ux-2*(b-a)*u/y**3 + (a-2*b)*uy/y**2 - b*uyy/y)
pdsolve(eq)

```

    0.1 0.0

    f(x, y) == F(-3*a*x + y**3)/(y**3)**(2/3)

```python
import scipy as sp
import time

gam = 0.1
beta = 0.25*gam*0.05
x0 = 5.0
dx = 0.1  #step size
dt = 10. #0.005
#dt =0.9/beta*1.0*dx**2
time_step=2000
nx = int(10/dx) #Lx = 2  x=[1,11]
dx2 = dx**2

#initial condition
xs = 1.0; xe= 11.0
xgrid = xs+np.arange(nx)*dx
#print xgrid
ui = sp.zeros([nx])
u = sp.zeros([nx])
dxi = 1.0/dx
dx2i = 1.0/dx2

ui = exp(-(xgrid-x0)**2)
plt.plot(xgrid,ui,'g-')
plt.ylim([-1,3.5])
plt.xlim([1,11])
#for i in range(nx):
#  if (np.abs(xgrid[i]-x0)<10*dx):
#    ui[i] = 2.5

c1 = 2.0*(beta-gam)
c2 = gam - 2.*beta
c3 = beta
#print np.arange(0,nx)
# differentiate the pde
def evolve_ts(u,ui):
#  for i in np.arange(1,nx-1):
#    ux = (u[i+1]-u[i-1])*dxi*0.5
#    uxx = (u[i+1]-2.0*u[i]+u[i-1])*dx2i
#  u[i] = 0.5*(ui[i-1]+ui[i+1])+dt*(c1*ui[i]+c2*ux+c3*uxx)
  u[1:-1] = 0.5*(ui[:-2]+ui[2:])+dt*(c1*ui[1:-1]/(xgrid[1:-1])**3 \
                                  + c2*(ui[2:]-ui[:-2])*dxi*0.5/(xgrid[1:-1])**2 \
                                  + c3*(ui[2:]-2.0*ui[1:-1]+ui[:-2])*dx2i/xgrid[1:-1])

#def updatefig(*args):
#  global u,ui

#time integration
for j in range(1,time_step):
  evolve_ts(u,ui)
  #if (j== time_step/10):
  #  plt.plot(xgrid,u)
  #  plt.ylim([-1,2.5])
  #  plt.xlim([1,11])

plt.plot(xgrid,u,'or',mfc='None')
```

test with advection:  $\partial_t y = \gamma\partial_x y$

```python
import scipy as sp
import time

gam = 1

dt = 0.1 #0.005
#dt =0.9/beta*1.0*dx**2
time_step=200000
xs = 1.0; xe= 11.0; x0=5
dx = 0.1  #step size
nx = int((xe-xs)/dx) #Lx = 2  x=[1,11]
dx2 = dx**2

#initial condition
xgrid = xs+np.arange(nx)*dx
#print xgrid
ui = sp.zeros([nx])
u = sp.zeros([nx])
dxi = 1.0/dx
dx2i = 1.0/dx2

ui = exp(-(xgrid-x0)**2)
plt.plot(xgrid,ui,'g-')
plt.ylim([-1,3.5])
plt.xlim([1,11])

#print np.arange(0,nx)
# differentiate the pde
def evolve_ts(u,ui):
#  for i in np.arange(1,nx-1):
#    ux = (u[i+1]-u[i-1])*dxi*0.5
#    uxx = (u[i+1]-2.0*u[i]+u[i-1])*dx2i
#  u[i] = 0.5*(ui[i-1]+ui[i+1])+dt*(c1*ui[i]+c2*ux+c3*uxx)
  u[1:-1] = 0.5*(ui[0:-2]+ui[2:])+dt*(gam*(ui[2:]-ui[0:-2])*dxi*0.5) #/(xgrid[1:-1])**2 \
#                                  + c3*(ui[2:]-2.0*ui[1:-1]+ui[:-2])*dx2i/xgrid[1:-1])

#def updatefig(*args):
#  global u,ui

#time integration
for j in range(1,time_step):
  global u,ui
  evolve_ts(u,ui)
  #if (j== time_step/10):
  #  plt.plot(xgrid,u)
  #  plt.ylim([-1,2.5])
  #  plt.xlim([1,11])

plt.plot(xgrid,u,'or',mfc='None')
```

# solve diffusion equation with FiPy

$\frac{\partial y}{\partial t} = D\frac{\partial^2 y}{\partial x^2} $

```python
%matplotlib inline
from fipy import *
nx = 50
dx = 1.
mesh = Grid1D(nx=nx, dx=dx)
L = 50
x=mesh.cellCenters[0]
phi = CellVariable(name="solution variable",  mesh=mesh, value=0.)
phi.setValue(1., where=(x > L/2. - L/10.) & (x < L/2. + L/10.))
D = 1.
#valueLeft = 0
#valueRight = 0
#phi.constrain(valueRight, mesh.facesRight)
#phi.constrain(valueLeft, mesh.facesLeft)
eqI = TransientTerm() == ExplicitDiffusionTerm(coeff=D)
timeStepDuration = 0.9 * dx**2 / (2 * D)
steps = 10
phiAnalytical = CellVariable(name="analytical value", mesh=mesh)

if __name__ == '__main__':
  viewer = Viewer(vars=(phi, phiAnalytical),datamin=0., datamax=1.)
  viewer.plot()
  x = mesh.cellCenters[0]
  t = timeStepDuration * steps
try:
  from scipy.special import erf
  phiAnalytical.setValue(1 - erf(x / (2 * numerix.sqrt(D * t))))
except ImportError:
  print "The SciPy library is not available to test the solution to \
  the transient diffusion equation"

for step in range(steps):
  eqI.solve(var=phi,dt=timeStepDuration)
  if __name__ == '__main__':
    viewer.plot()

print phi.allclose(phiAnalytical, atol = 7e-4)
if __name__ == '__main__':
   raw_input("Explicit transient diffusion. Press <return> to proceed...")
```

# solve advection equation with FiPy

$\frac{\partial y}{\partial t} = -v \frac{\partial y}{\partial x}$

```python
%matplotlib
from fipy import *
nx = 100
dx = 0.1
mesh = Grid1D(nx=nx, dx=dx)
L = nx*dx
x=mesh.cellCenters[0]
phi = CellVariable(name="solution variable",  mesh=mesh, value=0.)
#phi.setValue(1., where=(x > L/2. - L/10.) & (x < L/2. + L/10.))
phi.setValue(numerix.exp(-(x-L/4.)**2/0.1))

D = (1,)
#valueLeft = 0
#valueRight = 0
#phi.constrain(valueRight, mesh.facesRight)
#phi.constrain(valueLeft, mesh.facesLeft)
eqI = TransientTerm() + ExponentialConvectionTerm(coeff=D) == 0 #ConvectionTerm(coeff=D)
timeStepDuration = 0.1 * dx/D[0]
steps = 200
phiAnalytical = CellVariable(name="analytical value", mesh=mesh)

if __name__ == '__main__':
  viewer = Viewer(vars=(phi),datamin=0., datamax=1.)
  viewer.plot()
  x = mesh.cellCenters[0]
  t = timeStepDuration * steps

for step in range(steps):
  eqI.solve(var=phi,dt=timeStepDuration)
  if __name__ == '__main__':
    viewer.plot()

print phi.allclose(phiAnalytical, atol = 7e-4)
if __name__ == '__main__':
   raw_input("Explicit transient diffusion. Press <return> to proceed...")
```

    Using matplotlib backend: Qt4Agg
    False
    Explicit transient diffusion. Press <return> to proceed...

# solve the drift-diffuse equation with FiPy

$\partial_t y = \gamma \partial_x(x^{-2}y) + \beta\partial^2_x(x^{-1}y)$,

* make variable transfer by defining $x^{-2}y \rightarrow \phi$, as $x=(r/r_0)^{1/2}$, the end result of particle distribution of a function of radius is $\phi x^2$ vs. $x^2$.

* Units: $r_0 = 100$ AU ; $\Omega_0 = \frac{2\pi}{1000 yr}$

* After transfer, the equation reads
  $\partial_{\tau} (x^2\phi) = (\gamma+\beta) \partial_x y + \beta\partial^2_x y$,
  where $\gamma = h \tau_s (\tau_s^2+1)^{-1}$ measures the drift, $\beta = h \tilde{D}_{p,x}/4$ measures the
  diffusion    term, $h=H/R=0.1$, $\tilde{D}_{p,x} = D_{p,x}\Omega/c_s^2$.

* in the following calculation, we choose $\tau_s=1$(which results in the largest drift speed), timestep=$2\pi\Omega^{-1}$ (local orbits), duration of $100\times$ orbits $\sim 10^5$ yrs.

* For comparison, we calc and plot particle distribution after $10^5$ yrs due to drift, diffusion, and both.

```python
%matplotlib inline
#switch back to interactive instead of inline mode
from fipy import *
nx = 600
dx = 0.025
r_int = 1
L = nx*dx
mesh = Grid1D(nx=nx, dx=dx)+r_int
x = mesh.cellCenters[0]
x0 = 10.; sigma=0.05; fnorm = 1.0 #numerix.sqrt(2.*np.pi)*sigma
#print sigma,fnorm
##phi = P/j^2,  x= j= (r/r0)^{1/2}
phi_advc = CellVariable(name="drift",  mesh=mesh, value=0.)
phi_advc.setValue(numerix.exp(-(x-x0)**2/2./sigma**2)/fnorm/x**2, where=(x>x0-L/3) & (x<x0+L/3))

phi_diff = CellVariable(name="diffusion",  mesh=mesh, value=0.)
phi_diff.setValue(numerix.exp(-(x-x0)**2/2./sigma**2)/fnorm/x**2, where=(x>x0-L/3) & (x<x0+L/3))

phi_tot = CellVariable(name="drift+diffusion",  mesh=mesh, value=0.)
#phi.setValue(1., where=(x > L/2. - L/20.) & (x < L/2. + L/20.))
phi_tot.setValue(numerix.exp(-(x-x0)**2/2./sigma**2)/fnorm/x**2, where=(x>x0-L/3) & (x<x0+L/3))

taus=1.0
hor = 0.1
dp  = 0.05
gam  = hor*(taus/(1+taus**2))
beta = 0.25*hor*dp
valueLeft = 0
#valueRight = 0
#phi.constrain(valueRight, mesh.facesRight)
phi_advc.constrain(valueLeft, mesh.facesLeft)
phi_diff.constrain(valueLeft, mesh.facesLeft)
phi_tot.constrain(valueLeft, mesh.facesLeft)
print phi_advc*x**2

#vel = [0.+beta,] #[gam/X,]
#print vel
#D = beta

eq_advc = TransientTerm(coeff=x**2,var=phi_advc) == \
         ConvectionTerm(coeff=[gam,],var=phi_advc) #+ ExplicitDiffusionTerm(coeff=beta)
eq_diff = TransientTerm(coeff=x**2,var=phi_diff) == \
         ConvectionTerm(coeff=[beta,],var=phi_diff) + ImplicitDiffusionTerm(coeff=beta,var=phi_diff)
eq_tot =  TransientTerm(coeff=x**2,var=phi_tot)  == \
         ConvectionTerm(coeff=[beta+gam,],var=phi_tot)  + ImplicitDiffusionTerm(coeff=beta,var=phi_tot)

timeStepDuration = 0.5*6.28  #9 * dx**2 / (2 * D)
steps = 100*2

if __name__ == '__main__':
  viewer = Viewer(vars=(phi_advc*x*x,phi_diff*x*x,phi_tot*x*x),fipy_viewer='matplotlib', xmin=8,xmax=12, datamin=0., datamax=1.1)
  viewer.plot()
#  x = mesh.cellCenters()
#  t = timeStepDuration * steps

for step in range(steps):
  eq_advc.solve(var=phi_advc,dt=timeStepDuration)
  eq_diff.solve(var=phi_diff,dt=timeStepDuration)
  eq_tot.solve(var=phi_tot,dt=timeStepDuration)
  if __name__ == '__main__':
    viewer.plot()
print x
print phi_advc*x**2
print phi_diff*x**2
print phi_tot*x**2
```

```python

```

```python

```


**Submission deadline:**
* **Regular problems: last lab session before or on Monday, 9.12.**
* **Bonus problems: deadline for Lab Assignment 4**

However, because the list is published late, you get extra 7 late days *for the problems in this list only*.

**Points: 11 + 5 bonus points**

Please note: some of the assignments are tedious or boring if you are already a NumPy ninja. The bonus problems were designed to give you a more satisfying alternative.

The assignment is in the form of a Jupyter notebook. We will be using [Google Colab](https://colab.research.google.com) to solve it. Below you will find a "Setup" section. Follow instructions from this paragraph to download the notebook and open it using [Google Colab](https://colab.research.google.com).

Your goal is to solve problems posted below. Whenever possible, add your solutions to the notebook.

Please email us about any problems with it - we will try to correct them quickly. Also, please do not hesitate to use GitHub’s pull requests to send us corrections!

## Heads Up!

This assignment comes with starter code, but you are not forced to use it, as long as you execute all analysis demanded in the problems.

```python
# Please note that this code needs only to be run in a fresh runtime.
# However, it can be rerun afterwards too.
!pip install -q gdown httpimport

![ -e mnist.npz ] || gdown 'https://drive.google.com/uc?id=1QPaC3IKB_5tX6yIZgRgkpcqFrfVqPTXU' -O mnist.npz
```

    Downloading...
    From: https://drive.google.com/uc?id=1QPaC3IKB_5tX6yIZgRgkpcqFrfVqPTXU
    To: /Users/febrin/Desktop/Machine-Learning/ML uni course/Assignment3/mnist.npz
    55.4MB [01:31, 608kB/s]

```python
# Standard IPython notebook imports
%matplotlib inline

import os

from io import StringIO
import itertools
import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook

import scipy.stats as sstats
import scipy.optimize as sopt

import seaborn as sns
import sklearn.tree
import sklearn.ensemble
from scipy.linalg import solve_triangular

import graphviz
from numpy.linalg import inv

# In this way we can import functions straight from github
with httpimport.github_repo('janchorowski', 'nn_assignments',
                            module='common', branch='nn18'):
     from common.plotting import plot_mat

sns.set_style('whitegrid')
```

# Problem 1 [2p] Linear Regression

1. Implement a function generating a dataset of $n$ points
  according to the following algorithm:
  1. Draw $n$ points $x \propto U(0;10)$ (uniformly distributed on $[0,10]$).
  2. Draw $n$ points $y \propto \mathcal{N}(1+20x-1.3x^2, 7)$
    (from a Gaussian distribution with $\mu=1+20x-1.3x^2$ and $\sigma=7$).

  Prepare a dataset of 30 elements and make a scatterplot of
  the expected value $y$ in function $x$.

2. Use linear regression to fit polynomials to the
  generated dataset. Fit polynomials of degrees zero (a constant line),
  one, two and three. An easy way to do it is to transform each data
  point $x$ into a vector of its powers $[1, x, x^2, \ldots, x^m]$.

  Plot the dataset and all regression curves on one figure.

  **Note:** The name _linear regression_ signifies that the
  hypothesis is linear with respect to parameters $\Theta$.
  However, the relationship between $x$ and $y$ is not constrained
  to a linear one. In this exercise it is a polynomial one.

```python
def make_dataset(N):
    X = np.random.uniform(0, 10, N)
    mu = 1 + 20*X - 1.3 * X ** 2
    sigma = 7
    Y = np.random.normal(mu, sigma, N)
    return X, Y

data = make_dataset(30)
plt.scatter(data[0], data[1])
```

```python
def powers_of_X(X, degree):
    powers = np.arange(degree + 1).reshape(-1,1)
    return X**powers

def compute_polynomial(X, Theta):
    XP = powers_of_X(X, len(Theta) - 1) # len(Theta) x N
#     print('XP: ',XP.shape)
#     print('T: ',Theta.shape)
    Y = XP * Theta
    Y = np.sum(Y, axis=0)
#     print('Y:', Y.shape)
    return Y

plot_x_space = np.linspace(0,10,100)
plt.scatter(data[0], data[1])
for degree in range(4):
    X = powers_of_X(data[0], degree) # Matrix d x N
    Y = data[1].reshape(1, -1)       # Matrix 1 x N
    Theta = inv(X.dot(X.T)).dot(X).dot(Y.T)
    plt.plot(plot_x_space, compute_polynomial(plot_x_space, Theta).ravel(),
         label="degree: %d" %(degree, ))
    print(Theta, Theta.shape)
plt.legend(loc='lower right')
```

# Problem 2 [2p] Ridge Regression

When the data set is small and highly dimensional (or when high degree polynomials are used) the linear regression solution may fit the noise in the data instead of capturing the general rule. We call this phenomenon overfitting and will discuss it in detail in a few lectures.

One way of preventing overfitting is to force the model's parameters to be small. We call this *regularization*. Consider the following cost function:

$$ J(\Theta) = \frac{1}{2}\sum_{i=1}^N (y^{(i)} - \Theta^T x^{(i)})^2 + \frac{\lambda}{2} \Theta^T \Theta $$

Analyze datasets sampled using the following procedure:

1. $x \propto U(0;1)$: $x$ is sampled uniformly from the  $0-1$ range.
2. $y \propto \mathcal{N}(\mu=1+2x-5x^2 + 4x^3, \sigma=0.1)$: then
    $y$ is sampled from the Normal distribution with mean
    $\mu=1+2x-5x^2+4x^3$ and standard deviation $0.1$

Repeat 30 times an experiment in which you sample a new training
dataset, then fit polynomials of degree 0 to 14 and use $\lambda$
value from the set $\{0, 10^{-6}, 10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}\}$.

Plot the mean training and testing errors. What is the effect of increasing $\lambda$?

Plot how the error rates depend on the the polynomial degree and regularization constant. Try to find the best value for lambda on the test set, explain the model behavoir for small lambdas and large lambdas.

Then use a small lambda for numerical stability (1e-6) and train the model on increasingly large training sets. Plot th etraining cureves (train and test error rates versus amount of training data)

```python
#
# The true polynomial relation:
# y(x) = 1 + 2x -5x^2 + 4x^3
#
# TODO: write down the proper coefficients
#
true_poly_theta = np.array([1, 2, -5, 4])[:, np.newaxis]

def make_dataset(N, theta=true_poly_theta, sigma=0.1):
    """ Sample a dataset """
    X = np.random.rand(N)
    Y_clean = compute_polynomial(X, theta)
    Y = Y_clean + np.random.randn(N) * sigma
    return X,Y

train_data = make_dataset(30)
XX = np.linspace(0,1,100)
YY = compute_polynomial(XX, true_poly_theta)
plt.scatter(train_data[0], train_data[1], label='train data', color='r')
plt.plot(XX, compute_polynomial(XX, true_poly_theta).ravel(), label='ground truth')
plt.legend(loc='upper left')
```

```python
#please note: lambda is a reserved keyword in python, thus we use _lambda
def poly_fit(data, degree, _lambda):
    "Fit a polynomial of a given degree and weight decay parameter _lambda"
    X = powers_of_X(data[0], degree) # Matrix d x N
    Y = data[1].reshape(1, -1)       # Matrix 1 x N
    #
    # TODO: implement the closed-form solution for Theta
    #
    # Please note that np.inv may be numerically unstable.
    # It is better to use np.linalg.solve or even a QR decomposition.
    #
    # (X.T x X + _lambda x I)^-1 x X.T x Y
    I = np.eye(degree + 1)
    q, r = np.linalg.qr(np.dot(X, X.T) + _lambda * I, mode='complete')
    _inv = np.dot(inv(r), q.T)
    Theta = np.dot(np.dot(_inv, X), Y.T)
    return Theta
```

```python
def cost_ridge(Theta, X, Y, _lambda):
    vals = compute_polynomial(X, Theta)
    return np.mean((vals - Y) ** 2, axis=0)
```

```python
num_test_samples = 100
num_train_samples = [30]
lambdas = [0.0, 0.000001, 0.00001, 0.0001, 0.001, 0.01, 0.1, 1.0]
degrees = range(15)
num_repetitions = 30

#sample a single test dataset for all experiments
test_data = make_dataset(num_test_samples)
results = []

for (repetition,
     num_train,
     _lambda,
     degree,) in itertools.product(
         range(num_repetitions),
         num_train_samples,
         lambdas,
         degrees):
    train_data = make_dataset(num_train)
    Theta = poly_fit(train_data, degree, _lambda)

    train_err = cost_ridge(Theta, train_data[0], train_data[1], _lambda)
    test_err = cost_ridge(Theta, test_data[0], test_data[1], _lambda)

    results.append({'repetition': repetition,
                    'num_train': num_train,
                    'lambda': _lambda,
                    'degree': degree,
                    'dataset': 'train',
                    'err_rate': train_err})
    results.append({'repetition': repetition,
                    'num_train': num_train,
                    'lambda': _lambda,
                    'degree': degree,
                    'dataset': 'test',
                    'err_rate': test_err})
results_df = pd.DataFrame(results)
results_df.sample(5)
```

      dataset
      degree
      err_rate
      lambda
      num_train
      repetition

      5758
      train
      14
      0.009717
      1.00000
      30
      23

      5108
      train
      4
      0.010801
      0.00001
      30
      21

      3962
      train
      1
      0.016466
      0.00100
      30
      16

      2369
      test
      14
      0.010383
      0.10000
      30
      9

      165
      test
      7
      0.010312
      0.01000
      30
      0

```python
results_df.describe()
```

      degree
      err_rate
      lambda
      num_train
      repetition

      count
      7200.000000
      7.200000e+03
      7200.000000
      7200.0
      7200.000000

      mean
      7.000000
      3.822612e+04
      0.138889
      30.0
      14.500000

      std
      4.320794
      3.242778e+06
      0.327089
      0.0
      8.656043

      min
      0.000000
      1.816581e-03
      0.000000
      30.0
      0.000000

      25%
      3.000000
      9.245237e-03
      0.000008
      30.0
      7.000000

      50%
      7.000000
      1.114170e-02
      0.000550
      30.0
      14.500000

      75%
      11.000000
      1.545036e-02
      0.032500
      30.0
      22.000000

      max
      14.000000
      2.751588e+08
      1.000000
      30.0
      29.000000

```python
train = results_df[results_df['dataset'] == 'train']
test = results_df[results_df['dataset'] == 'test']
```

```python
%%time
#
# Plot how the error rates depend on the the polynomial degree and regularization
# constant.
# Try to find the best value for lambda on the test set, explain the model
# behavoir for small lambdas and large lambdas.
#
fig, ax = plt.subplots(1, 2, figsize=(15, 5))

ax[0].set_ylim([0, 0.1])
ax[1].set_ylim([0, 0.1])
ax[0].set_title('dataset = train')
ax[1].set_title('dataset = test')

sns.lineplot(x=train['degree'], y=train['err_rate'],
             hue=train['lambda'],
             palette=sns.color_palette("hls", len(lambdas)),
             ax=ax[0])

sns.lineplot(x=test['degree'], y=test['err_rate'],
             hue=test['lambda'],
             palette=sns.color_palette("hls", len(lambdas)),
             ax=ax[1])

plt.suptitle('Err rates for various lambdas and polynomial degree')
plt.show()
```

```python
fig, ax = plt.subplots(3, 3, figsize=(15, 15))

for row in range(3):
    for col in range(3):
        index = row*3 + col
        if index >= len(lambdas):
            break
        _lambda = lambdas[index]
        df = results_df[results_df['lambda'] == _lambda]

        ax[row][col].set_ylim([0, 0.1])
        ax[row][col].set_title(f'lambda = {_lambda}')

        sns.lineplot(x=df['degree'], y=df['err_rate'],
                     hue=df['dataset'],
                     ax=ax[row][col])

plt.suptitle('Train and test errors for various lambdas')
plt.show()
```

```python
%%time
# generate new data
num_test_samples = 100
num_train_samples = np.arange(10, 1001, 10)
lambdas = [0.000001]
degrees = np.arange(15)
num_repetitions = 30

#sample a single test dataset for all experiments
test_data = make_dataset(num_test_samples)
results = []

for (repetition,
     num_train,
     _lambda,
     degree,) in itertools.product(
         range(num_repetitions),
         num_train_samples,
         lambdas,
         degrees):
    train_data = make_dataset(num_train)
    Theta = poly_fit(train_data, degree, _lambda)

    train_err = cost_ridge(Theta, train_data[0], train_data[1], _lambda)
    test_err = cost_ridge(Theta, test_data[0], test_data[1], _lambda)

    results.append({'repetition': repetition,
                    'num_train': num_train,
                    'lambda': _lambda,
                    'degree': degree,
                    'dataset': 'train',
                    'err_rate': train_err})
    results.append({'repetition': repetition,
                    'num_train': num_train,
                    'lambda': _lambda,
                    'degree': degree,
                    'dataset': 'test',
                    'err_rate': test_err})
results_df = pd.DataFrame(results)
results_df.head()
```

    Wall time: 25.5 s

      dataset
      degree
      err_rate
      lambda
      num_train
      repetition

      0
      train
      0
      0.024578
      0.000001
      10
      0

      1
      test
      0
      0.066617
      0.000001
      10
      0

      2
      train
      1
      0.003266
      0.000001
      10
      0

      3
      test
      1
      0.026127
      0.000001
      10
      0

      4
      train
      2
      0.005662
      0.000001
      10
      0

```python
%%time
#
# Now set a small regularizatoin for numerical stability  (e.g. lambda=1e-6)
# and present the relationship between
# train and test error rates for varous degrees of the polynomial for
# different sizes of the train set.
#
_lambda = 1e-6
fig, ax = plt.subplots(5, 3, figsize=(15, 25))

for row in range(5):
    for col in range(3):
        degree = 3*row + col
        df = results_df[results_df['degree'] == degree]
        df = df[df['lambda'] == _lambda]

        ax[row][col].set_ylim([0, 0.1])
        ax[row][col].set_title(f'degree = {degree}')

        sns.lineplot(x=df['num_train'], y=df['err_rate'],
                     hue=df['dataset'],
                     ax=ax[row][col])

plt.suptitle('Err rates for various lambdas and polynomial degree')
plt.show()
```

# Problem 3 [1p + 2bp] Numerical optimization

Implement the [Rosenborck function](https://en.wikipedia.org/wiki/Rosenbrock_function). Then find its optimum using the [`scipy.optimize.fmin_l_bfgs_b`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html).

```python
#
# Implement the Rosenbrock function
#

with httpimport.github_repo('janchorowski', 'nn_assignments',
                            module='common', branch='nn18'):
    from common.gradients import check_gradient

def rosenbrock_v(P):
    """Returns the value of Rosenbrock's function at x"""
    a, b = 1.0, 100.0
    x, y = P[0], P[1]
    return (b * (y - x ** 2.0) ** 2.0 \
                  + (a - x) ** 2.0)

def rosenbrock(P):
    """
    Returns the value of rosenbrock's function and its gradient at x
    """
    val = rosenbrock_v(P)
    # Gradient should be np.array
    x, y = P
    dx = 400 * x ** 3 - 400 * x * y + 2 * x - 2
    dy = 200 * (y - x ** 2)
    dVdX = np.array([dx, dy])
    return [val, dVdX]

#
# Feel free to add your own test points.
#
for test_point in [[0., 0.],
                   [1., 1.],
                   [0.5, 1.0],
                   [1.0, 0.5]]:
    assert check_gradient(rosenbrock, np.array(test_point), prec=1e-5)
```

```python
#
# Use scipy.optimize.fmin_l_bfgs_b
# Make a contour plot of the RosenB=brock function, plot the optimizatoin path
# on the coutnour plot
#
# Hint: to save the points you can use the callback argument!
#

import scipy.optimize as sopt
lbfsg_hist = []
def save_hist(x):
    lbfsg_hist.append(np.array(x))

x_start = [0.,2.]
lbfsgb_ret = sopt.fmin_l_bfgs_b(rosenbrock, x_start, callback=save_hist)

# TODO: plot the countours of the function and overlay the optimization trajectory
lbfsg_hist = np.array(lbfsg_hist)
plt.plot(lbfsg_hist[:, 0], lbfsg_hist[:, 1])

mesh_x, mesh_y = np.meshgrid(np.linspace(0, 2, 100), np.linspace(0, 2, 100))
# print(mesh_x.shape, mesh_y.shape)
mesh = np.array([mesh_x, mesh_y])
# print(mesh.shape)
Z = np.array(rosenbrock_v(mesh))

plt.contour(mesh_x, mesh_y, Z, levels=100)
```

```python
def wolfe_conditions(f, xk, step_size, c1 = 0.1, c2 = 0.9):
    '''
    https://en.wikipedia.org/wiki/Wolfe_conditions?fbclid=IwAR2bebHWf_14B24J3L4eovZcK_q8yZidN5A9VEWS3XrDdSgHx8pxk8l5BNY
    '''
    f_xk,grad,H = f(xk)
    pk = -np.dot(qr_inv2(H),grad)
    f_xk_sspk,grad_xk_sspk,_ = f(xk + step_size*pk)
    if f_xk_sspk <= f_xk + c1*step_size*np.dot(pk.T,grad):
        if -np.dot(pk.T,grad_xk_sspk) <= -c2*np.dot(pk.T,grad):
            return True
    return False
```

```python
def rosenbrock_hessian(P):
    #TODO: compute the value, gradient and Hessian of Rosenbrock's function'
    val, D = rosenbrock(P)
    dvdx0, dvdx1 = D

    a, b = 1.0, 100.0
    x, y = P[0], P[1]

    h11 = 1200 * x ** 2 - 400 * y + 2
    h12 = -400 * x
    h21 = -400 * x
    h22 = 200
    H = np.array([[h11, h12], [h21, h22]])

    return [val, np.array((dvdx0, dvdx1)), H]
```

```python
def qr_inv2(a):
    q,r = np.linalg.qr(a, mode='complete')
    N = r.shape[0]
    inv_ = np.dot(solve_triangular(r,np.eye(N)),q.T)
    return inv_
```

```python
#
# Bonus problem
#

# Newtod-Raphson Method
# https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization?fbclid=IwAR3KYgYCwAagBCs8nU05jkole5z50FP0-IDsmi11fE1lf0mbTl6zqiogNzM

def Newton(f, position, alpha, stop_tolerance=1e-10, max_steps=1000000):

    # TODO:
    #  - implement the newton method and a simple line search
    #  - make sure your function is resilient at critical points (such as seddle points)
    #  - if the Newton direction is not minimizing the function, use the gradient for a few steps
    #  - try to beat L-BFGS on the bmber of function evaluations needed!

    history = [list(position)]
    fun_evals = 0

    for _ in range(int(max_steps)):
        value, gradient, Hessian = f(position)

        if abs(gradient[0]) < stop_tolerance and abs(gradient[1]) < stop_tolerance:
            break

        changed = False
        for try_step in range(10):
            step_size = alpha / (2 ** try_step)
            print(f"Tried step length: {step_size}")
            fun_evals += 1
            if wolfe_conditions(f, position, step_size=step_size):
                changed = True
                print(f"Accepted step length: {step_size}")
                position = position - step_size * np.dot(qr_inv2(Hessian),gradient)
                history.append(list(position))
                break
        if not changed:
            print(f"No step accepted, running L-BFGS for 3 steps")
            position = sopt.fmin_l_bfgs_b(rosenbrock, position, maxiter=3)[0]

    return position, history, fun_evals

Xopt, Xhist, fun_evals = Newton(rosenbrock_hessian, x_start, alpha=1e-0, stop_tolerance=1e-10, max_steps=1e6)
Xhist_ = np.array(Xhist)

print("Found optimum at %s in %d steps (%d function evals)(true minimum is at [1,1])" % (Xopt, len(Xhist), fun_evals))

MX,MY = np.meshgrid(np.linspace(-1,2,100), np.linspace(-1,2,100))
Z = np.array([MX,MY]).reshape(2,-1)
VR = rosenbrock_v(Z)
plt.contour(MX,MY,VR.reshape(MX.shape), 100)
plt.plot(Xhist_[:,0], Xhist_[:,1], '*-k')
```

# Problem 4 (Logistic Regression) [1p]

Linear regression is suitable for problems, where
the forecasted values are real numbers. We use logistic regression,
when we want to label the data with $0$ and $1$.

Let $x\in \mathbb{R}^n$ be a vector of $n$ real numbers, and
$y\in \{0,1\}$ the given class label. Similarly to what was shown
during the lecture, we add an additional element $x_0=1$
to vector $x$, to account for the bias term (and simplify the equations).

Similarly to linear regression, vector
$\Theta\in \mathbb{R}^{n+1}$ parametrizes the model
($n$ coefficients describes the data, the remaining one is the intercept).
In logistic regression, we model conditional probability that
sample $x$ belongs to class $1$ as:

\begin{equation}
p(\text{class}=1|x, \Theta)=h_\Theta(x) = \sigma\left(\sum_{j=0}^n \Theta_j x_j\right)
= \sigma\left(\Theta^T x \right),
\end{equation}

where $\sigma(a) = \frac{1}{1+\exp(-a)}$ is being called the logistic sigmoid
(a function, which plot is s-curved).

An unknown sample $x$ is being labeled $1$ if
$h_\Theta(x)\geq 0.5$, or equivalently, $\Theta^T x \geq 0$.

Classification mismatch between the forecasted values and
the data is being measured most of the time with cross-entropy:

\begin{equation}
    J(\Theta) = - \sum_{i=1}^m y^{(i)} \log \left(h_\Theta (x^{(i)})\right) + (1-y^{(i)}) \log \left(1-h_\Theta (x^{(i)})\right),
\end{equation}

assuming $0\log(0)=0$.

Use logistic regression to distinguish
  _Versicolor_ and _Virginica_ irises. Use only the
  `petal length` and `petal width` features. Use either
  Gradient Descent, or L-BFGS to solve for the optimal $\Theta$.
  Prepare the scatterplot of the data and plot the class separation
  boundary found by logistic regression.

```python
from sklearn import datasets
iris = datasets.load_iris()
print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)
petal_length = iris.data[:,iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]
```

    Features:  ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
    Targets:  ['setosa' 'versicolor' 'virginica']

```python
#Extract the petal_length and petal_width of versicolors and virginicas

IrisX = np.vstack([np.ones_like(petal_length), petal_length, petal_width])
IrisX = IrisX[:, iris.target!=0]

# Set versicolor=0 and virginia=1
IrisY = (iris.target[iris.target!=0]-1).reshape(1,-1).astype(np.float64)

plt.scatter(IrisX[1,:], IrisX[2,:], c=IrisY.ravel(), cmap='spring')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
```

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

```python
def logreg_loss(Theta, X, Y):
    #
    # Write a logistic regression cost suitable for use with fmin_l_bfgs
    #

    #reshape Theta
    ThetaR = Theta.reshape(X.shape[0],1)
    Z = Theta.T.dot(X)

    nll = -np.sum([
                    y * np.log2(sigmoid(Z)) \
                    + (1-y) * np.log2(1 - sigmoid(Z)) \
                    for y in Y
                    ])

    grad = X.dot((sigmoid(Z) - Y).T)

    #reshape grad into the shape of Theta, for fmin_l_bfsgb to work
    return nll, grad.reshape(Theta.shape)

Theta0 = np.zeros((3))

#
# Call a solver
#
ThetaOpt = sopt.fmin_l_bfgs_b(lambda Theta: logreg_loss(Theta, IrisX, IrisY), np.array(Theta0))[0]

#
# Now plot the found separation line
#

plt.scatter(IrisX[1,:], IrisX[2,:], c=IrisY.ravel(), cmap='spring')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
pl_min, pl_max = plt.xlim()
pl = np.linspace(pl_min, pl_max, 1000)
plt.plot(pl, -(ThetaOpt[0]+ThetaOpt[1]*pl)/ThetaOpt[2])
plt.xlim(pl_min, pl_max)
```

# Problem 5 [3p]

  The least squares method results in estimates that approximate
  the conditional mean of the response variable given certain values
  of the predictor variables.  However, for many applications we are interested
  in a median or other percentile rather than the mean of the response
  variable. An approximation of a percentile can be achieved
  indirectly by using the least squares method to fit a model,
  assuming a gaussian distribution on residuals of this model and
  calculating the required percentile of the gaussian distribution.
  The problems with this commonly used technique appear when the
  distribution of residuals does not follow a gaussian distribution.

  The quantile regression
  (https://en.wikipedia.org/wiki/Quantile_regression) aims at
  directly estimating a value of the conditional percentile of the
  response variable. It is often use in e.g. sales forecasting, where
  we are interested in e.g. keeping enough items to have a 90% chance
  to cover the demand. In quantile regression, instead of minimizing the mean squared error,
  quantile regression minimizes a different cost function namely:
  \begin{equation}
  f_{\tau} = \left\{
    \begin{array}{rl}
      \tau\cdot x & \text{if } x \geq 0,\\
      -(1-\tau)\cdot x & \text{if } x < 0,
    \end{array} \right.
  \end{equation}
  where $\tau$ is the precentile of interest.

Download the house pricing data set from
    https://raw.githubusercontent.com/janchorowski/nn_assignments/nn18/assignment3/03-house-prices-outliers.csv. To
    load it into Python, you can use the `pandas.read_csv`
    function, similarly to how we read the height/weight table in the
    notebook for lecture 3.

    This data contains information about areas and prices of around
    5000 houses offered for sale. Imagine you want to buy a 60 squared
    meters flat. How much money do you need to have to be able to
    choose between $75\%$ of all flats offered for sale?

1. **[1.5p]** Fit a linear regression model to the dependance of a house price
    on its area using the least squares method. Fit a Gaussian
    distribution to the residuals and calculate the 75 percentile of
    this distribution (you can look it up in a percentile table, or
    use the `scipy.stats.norm.ppf` function). Check the goodness of fit
    of your model by calculating the ratio of prices below the
    estimated 75 percentile. Understand the reason of the quality of
    the model by plotting a histogram of residuals and noting that
    they do not seem to follow a Gaussian distribution.
    How many samples are within the confidence interval? Is it too large or too small?
2. **[1.5]** Fit a quantile regression model by minimizing the $f_{\tau}$
    function. To fit the model write a function returning the cost,
    and its derivative (you can assume that the derivative is 0 at the
    singular points).  Use the L-BFGS solver (without step size
    control the GD algorithm does not converge to good solutions). You
    can start the solver from the least squares solution. Check the
    goodness of fit as described above.

    Now you can calculate the 75 percentile price value for a 60
    squared meters flat using the more adequate model.

    **Note**: Technically, we can not use a gradient-based
    optimization method to minimize the $f_{\tau}$ loss because it
    doesn't have a derivative at $x=0$. One solution is to use a
    subgradient method
    https://en.wikipedia.org/wiki/Subderivative. It is also
    possible to use a smooth loss function -- common variants are the
    Huber loss (https://en.wikipedia.org/wiki/Huber_loss) or a
    variant of the smooth approximation to the absolute value
    $\text{abs}(x) \approx \sqrt{x^2+\epsilon}$ with a small
    $\epsilon$. However, just ignoring the singular points tends to
    work well in practice.

```python
import pandas as pd
import scipy.optimize as sopt
from scipy.stats import norm
```

```python
data = pd.read_csv(
    'https://raw.githubusercontent.com/janchorowski/nn_assignments/nn18/assignment3/03-house-prices-outliers.csv',
    index_col=0)
data.head()
```

      rooms
      area
      price

      0
      3
      66.80
      367.801

      1
      3
      55.03
      289.458

      2
      4
      70.80
      389.000

      3
      1
      29.02
      195.695

      4
      2
      45.76
      265.754

```python
X = np.vstack((np.ones_like(data.rooms), data.area))
X
```

    array([[ 1.  ,  1.  ,  1.  , ...,  1.  ,  1.  ,  1.  ],
           [66.8 , 55.03, 70.8 , ..., 46.7 , 50.41, 48.1 ]])

```python
Y = np.asarray(data.price)[None,:]
Y
```

    array([[367.801, 289.458, 389.   , ..., 266.   , 267.1  , 269.   ]])

```python
Theta = inv(X.dot(X.T)).dot(X).dot(Y.T)
x_space = np.linspace(0, np.max(X), 100)
pred_Y = compute_polynomial(x_space, Theta).ravel()
plt.plot(x_space, pred_Y, c='red')
plt.scatter(X[1, :], Y, edgecolors='black', s=30)
print('Theta:\n', Theta)
```

```python
# closer look
plt.plot(x_space, pred_Y, c='red')
plt.scatter(X[1, :], Y, edgecolors='black', s=30)
plt.ylim([0, 2000])
plt.xlim([0, 200])
```

```python
# get residuals and delete outliners
residuals = Y - compute_polynomial(X[1, :], Theta).ravel()
```

```python
# find mu and sigma
mu = residuals.mean()
sigma = np.sqrt(np.mean((residuals - mu) ** 2))
mu, sigma
```

    (5.40780633486606e-14, 629.5549979455304)

```python
# plot residuals
sns.distplot(residuals, bins=60, hist_kws={'edgecolor' : 'black'})
```

```python
# plot residuals
threshhold = 400
residuals = residuals[residuals < threshhold]
residuals = residuals[residuals > -threshhold]
sns.distplot(residuals, bins=60, hist_kws={'edgecolor' : 'black'})
plt.xlim([-500, 500])
```

```python
# get 75 percentile of distribution
# tells us that 75% of errors are < than 424.
sstats.norm.ppf(0.75, mu, sigma)
```

    424.6283932989756

```python
np.mean(residuals < sstats.norm.ppf(0.75, mu, sigma))
```

    0.9940878378378378

# Quantile regression

```python
def quantile_cost(Y, Theta, X, tau):
    '''
    Returns:
        pair (x, y), where
        x: function values mean
        y = (a, b)
        y: gradient mean
    '''

    residuals = Y - Theta.T.dot(X)
    residuals = residuals.ravel()
    ans = []

    for i, r in enumerate(residuals):
        if r == 0:
            val = np.array([0, np.array([0, 0])])
            ans.append(val)
        elif r > 0:
            val = np.array([np.array(tau * r),
                            np.array(-tau * X[:, i])])
            ans.append(val)
        else:
            val = np.array([np.array((tau - 1) * r),
                            np.array((1 - tau) * X[:, i])])
            ans.append(val)

    ans = np.array(ans)
    N = residuals.size
    func_val, gradient_vals = 0, np.array([0, 0], dtype=np.float64)

    for f_val, gradient_val in ans:
        func_val += f_val
        gradient_vals += gradient_val

    func_val /= N
    gradient_vals /= N
#     print(func_val, gradient_vals)

    return (func_val, gradient_vals)

def get_theta(X, Y, Theta, tau=0.75):
    hist = []
    def save_hist(x):
        hist.append(np.array(x))

    lbfsgb_ret = sopt.fmin_l_bfgs_b(lambda theta: quantile_cost(Y, theta, X, tau),
                                    Theta, callback=save_hist)

#     print(hist)
#     print(lbfsgb_ret)
    return hist[-1]
```

```python
%%time
taus = [0.25, 0.50, 0.75]
colors = ['red', 'green', 'blue']
x_space = np.linspace(0, np.max(X), 100)

for i, tau in enumerate(taus):
    theta = get_theta(X, Y, Theta, tau=tau)
    plt.plot(x_space, theta[0] + theta[1] * x_space,
             c=colors[i], label=tau)
    residuals = Y - compute_polynomial(X[1, :], theta[:, np.newaxis]).ravel()
    mu = residuals.mean()
    sigma = np.sqrt(np.mean((residuals - mu) ** 2))
    errors = residuals[residuals < 0].size / residuals.size

    print(f'Quant: {tau}\n{theta}')
    print(f'Fraction of negative errors: {errors}')
    print(f'mu: {mu}, sigma: {sigma}\n')

plt.legend(loc='lower right')
plt.scatter(X[1, :], Y, edgecolors='black', s=30, alpha=0.7)
plt.ylim([0, 2000])
plt.xlim([0, 200])
```

# Problem 6 [3p] Random Forests v2

Repeat Problem 4 from Assignment 2 using [SKlearn's Decision Trees](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html). Since they don't support categorical attributes, use the numerical variant of the German Credit dataset (cf. the AdaBoost demo at https://github.com/janchorowski/ml_uwr/blob/fall2019/lectures/05_adabost_demo.ipynb), or try to encode them using the [1-Hot Encoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html).

**NB:** If you did Assignment 2 Problem 4, you automatically get 2 points for this problem. We highly advise to not do this problem, but do a bonus one instead.

# Problem 7 [3bp] Lasso and Adaboost

Boosting is similar to Lasso regression: it greedily selects classifiers that lead to the largest decrease in the train error.

However, once boosting adds a tree to the ensemle, it doesn't revisit this decision. We can try to get a small improvement by trying to refit the parameters in the tree (the tree weights and distributions in leaves) using some global optimization algorithm, such as LASSO.

In this exercise:
1. Train a boosting classifier on some dataset (e.g. German Credit from Problem 6).
2. Then use the tree to build a new representation of data:
    1. Take each sample and put it through all trees.
    2. For each tree, record the leaf index.

   Thus, a sample is represented by the leaf ids. Encode them using 1-hot encoding and train a LASSO classifier on this new representation.

```python

```

    (300, 100, 1)

```python

```


# Introduction to PyCBC Inference 4: Advanced configuration settings
### Collin Capano

In this tutorial we delve into the more advanced settings for the configuration file. These can be used to fine-tune your run, and do more elaborate analyses.

We will use the provided example configuration file [bbh_example.ini](bbh_example.ini). This is the configuration file you would need to analyze signals like GW150914.

### Prerequisites

```python
import os
from IPython.display import Image
from IPython.core.display import HTML
```

## Sampling transforms (MCMC samplers only)

The parameter space in which a prior is most easily defined is not necessarily the best space for sampling. The most common example of this are the component masses. We typically use a uniform prior on each component mass, which is easy to define and evaluate. However, component masses are difficult to measure. In addition, if the prior is symmetric between the components (meaning you could swap the labels `1` and `2` and get the same result), then the posterior will be symmetric. This makes it difficult for the sampler to converge on the correct value.

Better parameters are the *chirp mass* $\mathcal{M}$ and *mass ratio* $q$. While the mass ratio is also difficult to measure (and in fact is less well-constrained than either component masses), chirp mass can be measured, especially for binary neutron star systems. It is therefore easier and faster for samplers to converge on chirp mass and mass ratio rather than `mass1` and `mass2`.

*Sampling transforms* provide the ability to replace one or more parameters in the `[variable_params]` section with other parameters that are (presumably) easier to sample. To use them, you must provide a `[sampling_params]` section. This lists the names of the parameters that will be used to sample in, in place of the variable params. In our example, we have:
```
[sampling_params]
; parameters on the left will be sampled in
; parametes on the right
mass1, mass2 : mchirp, q
```
In addition, we need to provide `[sampling_transforms]` sections that define what transform to use. Similar to the `[prior]` sections, the `[sampling_transforms]` must list the output parameter(s) in the header, as a `+` separated list. In our example we have:
```
[sampling_transforms-mchirp+q]
; inputs mass1, mass2
; outputs mchirp, q
name = mass1_mass2_to_mchirp_q
```
The `name` parameter defines which transform to use. Transforms are defined in PyCBC's [transforms module](https://pycbc.org/pycbc/latest/html/pycbc.html#module-pycbc.transforms); see the table [here](https://pycbc.org/pycbc/latest/html/inference.html#sampling-transforms) for a list of all of the currently available transforms.

In our case, we are using the [Mass1Mass2ToMchirpQ](https://pycbc.org/pycbc/latest/html/pycbc.html#pycbc.transforms.Mass1Mass2ToMchirpQ) transform. This knows to take the parameters named `mass1` and `mass2` and turn them into chirp mass and mass ratio, called `mchirp` and `q`, respectively. If you named your mass parameters something different, you can set them by adding `mass1_param` and `mass2_param`. For example, if your mass parameters were called `srcmass1` and `srcmass2`, you would do:
```
[sampling_params]
srcmass1, srcmass2 : mchirp, q

[sampling_transforms-mchirp+q]
name = mass1_mass2_to_mchirp_q
mass1_param = srcmass1
mass2_param = srcmass2
```

#### Notes:
 * The dimension of the sampling parameter space must be the same as the prior space. This means that the number of parameters replaced must equal the number of parameters sampled.
 * An inverse transform and Jacobian are needed in order to transform between the sampling parameter space and the prior space. These are given by the transforms' `inverse_transform` attribute and `jacobian` method, respectively. Not all transforms in the `transforms` module have these defined. If you try to use a transform that does not have these properties, you'll get a `NotImplementedError`.
 * **The nested samplers do not support sampling transforms.** This is due to the way that they sample the prior. At least for `dynesty`, we have found that it converges fine without needing to switch to chirp mass and mass ratio.

## Waveform transforms

**The name and number of parameters set in the `[variable_params]` section can be anything.** If you like, you could name a parameter `claxmanfooblefar`. (There is one exception: names that start with `calib_` are assumed to be calibration parameters. See [here](https://pycbc.org/pycbc/latest/html/inference.html#calibration-parameters) for details.) However, certain parameter names must be provided to the waveform generator in order for it to work properly. For example, to set the masses of the binary, it must get parameters named `mass1` and `mass2`.

The full list of recognized parameters by the CBC waveform generators can be found in the "Waveform transforms" section of the [PyCBC Inference documentation](https://pycbc.org/pycbc/latest/html/inference.html#waveform-transforms). Looking at the table there, we see that the spin parameters must be specified in cartesian coordinates, with names `spin1x`, `spin1y`, `spin1z`, `spin2x`, `spin2y`, `spin2z`. But looking at our list of `variable_params` we see that we've listed things called `spin1_a`, `spin1_polar`, `spin1_azimuthal` (same for `spin2`). Scrolling down to the prior section, we see that the priors for these parameters are:
```
[prior-spin1_a]
name = uniform
min-spin1_a = 0.0
max-spin1_a = 0.99

[prior-spin1_polar+spin1_azimuthal]
name = uniform_solidangle
polar-angle = spin1_polar
azimuthal-angle = spin1_azimuthal
```
(Likewise for `spin2`.) This defines a 3D prior that is uniform in magnitude and isotropic in direction. So, `spin1_a`, `spin1_polar` and `spin1_azimuthal` are spherical coordinates. But this doesn't mean anything to the waveform generator. It has no idea what something called `spin1_a` does, regardless of what its prior is.

To link our arbitrarily named `variable_params` to things that the waveform generator recognizes, we need to provide a `[waveform_transforms]` section. The syntax is the same as `[sampling_transforms]`:
 * The output of the transform needs to be provided in the section header. If it outputs multiple parameters, they should be `+` separated.
 * A `name` must be provided that gives the name of the transform to use.
 * Any transform in the [transforms module](https://pycbc.org/pycbc/latest/html/pycbc.html#module-pycbc.transforms) may be used.

In our example we have:
```
[waveform_transforms-spin1x+spin1y+spin1z]
name = spherical_to_cartesian
x = spin1x
y = spin1y
z = spin1z
radial = spin1_a
polar = spin1_polar
azimuthal = spin1_azimuthal
```
The `spherical_to_cartesian` transforms the spherical coordinates into cartesian. We have to tell the names of each of the cartesian and spherical components that is transforming to/from. We have added a similar section for spin2.

## Variable vs waveform vs sampling parameters

```python
Image('paramsdiagram.png')
```

## Custom waveform transforms

Unlike sampling transforms, waveform transforms do not need a Jacobian nor an inverse specified. In addition, the number of waveform parameters need not be the same as the number of `variable_params`. This means that you can use the [CustomTransform](http://pycbc.org/pycbc/latest/html/pycbc.html#pycbc.transforms.CustomTransform) to create arbitrary transforms using any function in the [pycbc.conversions](http://pycbc.org/pycbc/latest/html/pycbc.html#module-pycbc.conversions), [pycbc.coordinates](http://pycbc.org/pycbc/latest/html/pycbc.html#module-pycbc.coordinates), or [pycbc.cosmology](http://pycbc.org/pycbc/latest/html/pycbc.html#module-pycbc.cosmology) modules, along with numpy math functions.

For example, the included [gw170817_example-common_eos.ini](gw170817_example-common_eos.ini) configuration file is similar to what was used in [De et al.](https://arxiv.org/abs/1804.08583) to analyze GW170817 (a binary neutron star) with a common equation of state. The common equation of state was enforced by having a single `lambdasym` parameter to describe the tidal deformability. The tidal deformation of each component `lambda1` and `lambda2` was related to `lambdasym` via:

\begin{align}
\Lambda_1 &= q^3 \Lambda_{\mathrm{sym}} \\
\Lambda_2 &= q^{-3} \Lambda_{\mathrm{sym}}
\end{align}

That is accomplished in the configuration file via:
```
[waveform_transforms-lambda1]
name = custom
inputs = mass1, mass2, lambdasym
lambda1 = lambdasym/((mass1/mass2)**3)

[waveform_transforms-lambda2]
name = custom
inputs = mass1, mass2, lambdasym
lambda2 = lambdasym/((mass2/mass1)**3)
```

## Challenge:

Modify [bbh_example.ini](bbh_example.ini) to create an analysis that uses a prior uniform in *source* masses. Hints:
 * The waveform generator needs the masses in the detector frame. In other words, you have to use the distance to redshift the mass parameters that are passed to the waveform generator.
 * You will need to use waveform transforms.
 * There is a `redshift` function available in the cosmology module that takes distance as an argument.
 * Keep in mind that the parameters passed to the waveform generator must be called `mass1` and `mass2`. What will you call the mass parameters in the `variable_params` section and prior?

*The answer can be found in [bbh_example-src_masses.ini](bbh_example-src_masses.ini).*