Blankenbach Benchmark Case 1
======

Steady isoviscous thermal convection
----

Two-dimensional, incompressible, bottom heated, steady isoviscous thermal convection in a 1 x 1 box, see case 1 of Blankenbach *et al.* 1989 for details.

**This example introduces:**
1. Loading/Saving variables to disk.
2. Defining analysis tools.
3. Finding a steady state.

**Keywords:** Stokes system, advective diffusive systems, analysis tools

**References**

B. Blankenbach, F. Busse, U. Christensen, L. Cserepes, D. Gunkel, U. Hansen, H. Harder, G. Jarvis, M. Koch, G. Marquart, D. Moore, P. Olson, H. Schmeling and T. Schnaubelt. A benchmark comparison for mantle convection codes. Geophysical Journal International, 98, 1, 23–38, 1989
http://onlinelibrary.wiley.com/doi/10.1111/j.1365-246X.1989.tb05511.x/abstract

```python
import underworld as uw
from underworld import function as fn
import underworld.visualisation as vis
import math
import numpy as np

try:
    from xvfbwrapper import Xvfb
    vdisplay = Xvfb()
    vdisplay.start()
except:
    pass
```

Setup parameters
-----

```python
boxHeight = 1.0
boxLength = 1.0
# Set grid resolution.
res = 128
# Set max & min temperautres
tempMin = 0.0
tempMax = 1.0
```

Choose which Rayleigh number, see case 1 of Blankenbach *et al.* 1989 for details.

```python
case = "a"
if(case=="a"):
    Ra=1.e4
    eta0=1.e23
elif(case=="b"):
    Ra=1.e5
    eta0=1.e22
else:
    Ra=1.e6
    eta0=1.e21
```

Set input and output file directory

```python
inputPath  = 'input/1_03_BlankenbachBenchmark/'
outputPath = 'output/'
# Make output directory if necessary.
if uw.mpi.rank==0:
    import os
    if not os.path.exists(outputPath):
        os.makedirs(outputPath)
```

Create mesh and variables
------

```python
mesh = uw.mesh.FeMesh_Cartesian( elementType = ("Q1/dQ0"),
                                 elementRes  = (res, res),
                                 minCoord    = (0., 0.),
                                 maxCoord    = (boxLength, boxHeight))

velocityField       = mesh.add_variable(         nodeDofCount=2 )
pressureField       = mesh.subMesh.add_variable( nodeDofCount=1 )
temperatureField    = mesh.add_variable(         nodeDofCount=1 )
temperatureDotField = mesh.add_variable(         nodeDofCount=1 )

# initialise velocity, pressure and temperatureDot field
velocityField.data[:]       = [0.,0.]
pressureField.data[:]       = 0.
temperatureField.data[:]    = 0.
temperatureDotField.data[:] = 0.
```

Set up material parameters and functions
-----

Set values and functions for viscosity, density and buoyancy force.

```python
# Set a constant viscosity.
viscosity = 1.

# Create our density function.
densityFn = Ra * temperatureField

# Define our vertical unit vector using a python tuple (this will be automatically converted to a function).
z_hat = ( 0.0, 1.0 )

# A buoyancy function.
buoyancyFn = densityFn * z_hat
```

Set initial temperature field
-----
The initial temperature field can be loaded from a pre-run steady state data set ( ``LoadFromFile = True`` ) or set to a sinusodial perterbation ( ``LoadFromFile = False`` ).

```python
# Steady state temperature field to be loaded from data file.
LoadFromFile = True
```

**If loading steady state data set**

Data is stored in h5 format from a 64\*64 grid resolution model.  Data has been saved for 3 different Rayleigh numbers, $Ra = 10^4$, $10^5$ or $10^6$.

Once loaded the data will need to be re-meshed onto a new grid, unless the new resolution is also 64\*64.

For more information on using meshes see the user guide.

```python
if(LoadFromFile == True):
    # Setup mesh and temperature field for 64*64 data file.
    mesh64 = uw.mesh.FeMesh_Cartesian( elementType = ("Q1/dQ0"),
                                       elementRes  = (64, 64),
                                       minCoord    = (0., 0.),
                                       maxCoord    = (boxLength, boxHeight),
                                       partitioned = False )
    temperatureField64  = mesh64.add_variable( nodeDofCount=1 )

    # read in saved steady state temperature field data
    if( case == "a" ):
        temperatureField64.load(inputPath+'tempfield_inp_64_Ra1e4.h5')
        print('Loading 64*64 for Ra = 1e4')
    elif( case == "b" ):
        temperatureField64.load(inputPath+'tempfield_inp_64_Ra1e5.h5')
        print('Loading 64*64 for Ra = 1e5')
    else:
        temperatureField64.load(inputPath+'tempfield_inp_64_Ra1e6.h5')
        print('Loading 64*64 for Ra = 1e6')

    if( res==64 ): # no remeshing needed, copy directly
        temperatureField.data[:] = temperatureField64.data[:]
    else: # remeshing needed
        temperatureField.data[:] = temperatureField64.evaluate(mesh)
```

**If using sinusodial perturbation**

```python
if(LoadFromFile == False):
    temperatureField.data[:] = 0.
    pertStrength = 0.1
    deltaTemp = tempMax - tempMin
    for index, coord in enumerate(mesh.data):
        pertCoeff = math.cos( math.pi * coord[0]/boxLength ) * math.sin( math.pi * coord[1]/boxLength )
        temperatureField.data[index] = tempMin + deltaTemp*(boxHeight - coord[1]) + pertStrength * pertCoeff
        temperatureField.data[index] = max(tempMin, min(tempMax, temperatureField.data[index]))
```

**Show initial temperature field**

```python
fig = vis.Figure()
fig.append( vis.objects.Surface(mesh, temperatureField) )
fig.show()
```

Create boundary conditions
----------

Set temperature boundary conditions on the bottom ( ``MinJ`` ) and top ( ``MaxJ`` ).

```python
for index in mesh.specialSets["MinJ_VertexSet"]:
    temperatureField.data[index] = tempMax
for index in mesh.specialSets["MaxJ_VertexSet"]:
    temperatureField.data[index] = tempMin
```

Construct sets for the both horizontal and vertical walls. Combine the sets of vertices to make the ``I`` (left and right side walls) and ``J`` (top and bottom walls) sets.

```python
iWalls = mesh.specialSets["MinI_VertexSet"] + mesh.specialSets["MaxI_VertexSet"]
jWalls = mesh.specialSets["MinJ_VertexSet"] + mesh.specialSets["MaxJ_VertexSet"]

freeslipBC = uw.conditions.DirichletCondition( variable        = velocityField,
                                               indexSetsPerDof = (iWalls, jWalls) )
tempBC     = uw.conditions.DirichletCondition( variable        = temperatureField,
                                               indexSetsPerDof = (jWalls,) )
```

System setup
-----

**Setup a Stokes system**

```python
stokes = uw.systems.Stokes(    velocityField = velocityField,
                               pressureField = pressureField,
                               conditions    = [freeslipBC,],
                               fn_viscosity  = viscosity,
                               fn_bodyforce  = buoyancyFn )
# get the default stokes equation solver
solver = uw.systems.Solver( stokes )
```

**Create an advection diffusion system**

```python
advDiff = uw.systems.AdvectionDiffusion( phiField       = temperatureField,
                                         phiDotField    = temperatureDotField,
                                         velocityField  = velocityField,
                                         fn_diffusivity = 1.0,
                                         conditions     = [tempBC,] )
```

Analysis tools
-----

**Nusselt number**

The Nusselt number is the ratio between convective and conductive heat transfer

\\[
Nu = -h \frac{ \int_0^l \partial_z T (x, z=h) dx}{ \int_0^l T (x, z=0) dx}
\\]

```python
nuTop    = uw.utils.Integral( fn=temperatureField.fn_gradient[1],
                              mesh=mesh, integrationType='Surface',
                              surfaceIndexSet=mesh.specialSets["MaxJ_VertexSet"])

nuBottom = uw.utils.Integral( fn=temperatureField,
                              mesh=mesh, integrationType='Surface',
                              surfaceIndexSet=mesh.specialSets["MinJ_VertexSet"])
```

```python
nu = - nuTop.evaluate()[0]/nuBottom.evaluate()[0]
print('Nusselt number = {0:.6f}'.format(nu))
```

**RMS velocity**

The root mean squared velocity is defined by intergrating over the entire simulation domain via

\\[
\begin{aligned}
v_{rms}  =  \sqrt{ \frac{ \int_V (\mathbf{v}.\mathbf{v}) dV } {\int_V dV} }
\end{aligned}
\\]

where $V$ denotes the volume of the box.

```python
intVdotV = uw.utils.Integral( fn.math.dot( velocityField, velocityField ), mesh )

vrms = math.sqrt( intVdotV.evaluate()[0] )
print('Initial vrms = {0:.3f}'.format(vrms))
```

Main simulation loop
-----

If the initial conditions are loaded from file then this loop will only take a single step. If you would like to run the entire simulation from a small perturbation then change the ``LoadFromFile`` variable above to equal ``False``. Warning: the simulation will take a long time to get to steady state.

```python
#initialise time, step, output arrays
time = 0.
step = 0
timeVal = []
vrmsVal = []

# starting from steady state == True
if(LoadFromFile == True):
    step_end = 1
else:
    step_end = 5000

# output frequency
step_output = max(1,min(100, step_end/10))
epsilon    = 1.e-8

velplotmax = 0.0
nuLast     = -1.0
```

```python
# define an update function
def update():
    # Determining the maximum timestep for advancing the a-d system.
    dt = advDiff.get_max_dt()
    # Advect using this timestep size.
    advDiff.integrate(dt)
    return time+dt, step+1
```

```python
# Perform steps.
while step<=step_end:
    # Solving the Stokes system.
    solver.solve()

    # Calculate & store the RMS velocity and Nusselt number.
    vrms = math.sqrt( intVdotV.evaluate()[0] )
    nu   = - nuTop.evaluate()[0]/nuBottom.evaluate()[0]
    vrmsVal.append(vrms)
    timeVal.append(time)
    velplotmax = max(vrms, velplotmax)

    # print output statistics
    if step%(step_end/step_output) == 0:
        if(uw.mpi.rank==0):
            print('steps = {0:6d}; time = {1:.3e}; v_rms = {2:.3f}; Nu = {3:.3f}; Rel change = {4:.3e}'
              .format(step, time, vrms, nu, abs((nu - nuLast)/nu)))
    # Check loop break conditions.
    if(abs((nu - nuLast)/nu) < epsilon):
        if(uw.mpi.rank==0):
            print('steps = {0:6d}; time = {1:.3e}; v_rms = {2:.3f}; Nu = {3:.3f}; Rel change = {4:.3e}'
              .format(step, time, vrms, nu, abs((nu - nuLast)/nu)))
        break
    nuLast = nu

    # update
    time, step = update()
```

Post analysis
-----

**Benchmark values**

The time loop above outputs $v_{rms}$ and $Nu$ as general statistics for the system. For comparison, the benchmark values for the RMS velocity and Nusselt number are shown below for different Rayleigh numbers. All benchmark values shown below were determined in Blankenbach *et al.* 1989 by extroplation of numerical results.

| $Ra$          | $v_{rms}$   | $Nu$  |   $q_1$  | $q_2$  |
| ------------- |:-------------:|:-----:|:-----:|:-----:|
| 10$^4$ | 42.865 |  4.884 | 8.059 | 0.589 |
| 10$^5$ | 193.215 | 10.535 | 19.079 | 0.723 |
| 10$^6$ | 833.990 | 21.972 | 45.964 | 0.877 |

```python
# Let's add a test to ensure things are working as expected
if case == "a":
    if not np.isclose(nu,4.884,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected Nusselt number.")
    if not np.isclose(vrms,42.865,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected Nusselt number.")
```

**Resulting pressure field**

Use the same method as above to plot the new temperature field. This can also be used to plot the pressure field, or any other data structures of interest.

```python
figtemp = vis.Figure()
figtemp.append( vis.objects.Surface( mesh, pressureField ) )
figtemp.show()
```

**Plot the velocity vector field**

For this example the velocity field is interesting to see. This is visualised in two ways, firstly plotting a surface colour map of the velocity magnitude, and secondly the velocity vectors at points on the mesh. For aesthetics the vector arrows are scaled by a little more than the maximum $v_{rms}$ value found in the time loop above.

```python
fig2 = vis.Figure()
velmagfield = uw.function.math.sqrt( uw.function.math.dot( velocityField, velocityField ) )
fig2.append( vis.objects.VectorArrows(mesh, velocityField/(2.5*velplotmax), arrowHead=0.2, scaling=0.1) )
fig2.append( vis.objects.Surface(mesh, temperatureField) )
fig2.show()
```

Parallel friendly post analysis
----

When running underworld in parallel the data of each mesh variable is spread across all the processors. However often we will want to calculate a quantity based on data at specific points that may not all be on the same processor.

A solution is presented here which consists of saving the data from all processors to file, then reloading the mesh variable data using a new non-partitioned mesh. This enables all the data to be available to each processor. We will the carry out the post analysis using the first processor.

**Save temperature, pressure and velocity data**

Save the basic mesh variable data to files using the HDF5 format. This is the same file type as is loaded above.

```python
mesh.save(outputPath+"mesh.h5")
temperatureField.save(outputPath+'tempfield.h5')
pressureField.save(outputPath+'presfield.h5')
velocityField.save(outputPath+'velfield.h5')
```

**Construct new mesh and variable on non-partitioned mesh**

Read saved mesh variable data into a new mesh variable where the information is not partitioned across multiple processors. This means that we can use a single processor to access all the data and calculate some quantities of interest.

```python
# build a non-partitioned mesh with same box size
mesh0 = uw.mesh.FeMesh_Cartesian( elementType = ("Q1/dQ0"),
                                  elementRes  = (res, res),
                                  minCoord    = (0., 0.),
                                  maxCoord    = (boxLength, boxHeight),
                                  partitioned = False )
# load previous mesh coordinate data onto new non-partitioned mesh
mesh0.load(outputPath+'mesh.h5')

# load T, P and V data onto the new mesh
# note that pressure is always on the submesh
temperatureField0  = mesh0.add_variable( nodeDofCount=1 )
pressureField0     = mesh0.subMesh.add_variable( nodeDofCount=1 )
velocityField0     = mesh0.add_variable( nodeDofCount=2 )
temperatureField0.load(outputPath+"tempfield.h5")
pressureField0.load(outputPath+"presfield.h5")
velocityField0.load(outputPath+"velfield.h5")
```

**Temperature gradient**

The final benchmarks in the Blankenbach paper involve the temperature gradient in the vertical direction ($\frac{\partial T}{\partial z}$). This is easy to find using the underworld functions, as shown below.

```python
if(uw.mpi.rank==0):
    tempgradField = temperatureField0.fn_gradient
    vertTGradField = - boxHeight * tempgradField[1] / tempMax # scaled for direct benchmarking below
```

**More benchmark values**

The vertical temperature gradient (above) is set up to be non-dimensional as per Blankenbach et al 1989. To compare to the benchmark values in their work the gradient is compared at the corners of the simulation box: $q_1$ at $x = 0$, $z = h$; $q_2$ at $x = l$, $z = h$; $q_3$ at $x = l$, $z = 0$; $q_4$ at $x = 0$, $z = 0$. Where $h$ = Box_Height and $l$ = Box_Length and the non-dimensional gradient field is given by
\\[
q = \frac{-h}{\Delta T} \left( \frac{\partial T}{\partial z} \right)
\\]

Provided the simulation is run to steady-state with sufficent resolution then the $q$ values should be close to the benchmark values given again below for different Rayleigh numbers.

| $Ra$          |    $q_1$  | $q_2$  |
| ------------- |:-----:|:-----:|
| 10$^4$ | 8.059 | 0.589 |
| 10$^5$ | 19.079 | 0.723 |
| 10$^6$ | 45.964 | 0.877 |

```python
if(uw.mpi.rank==0):
    q1 = vertTGradField.evaluate( (0., boxHeight))[0][0]
    q2 = vertTGradField.evaluate( (boxLength, boxHeight))[0][0]
    q3 = vertTGradField.evaluate( (boxLength, 0.))[0][0]
    q4 = vertTGradField.evaluate( (0., 0.))[0][0]

    print('Rayleigh number = {0:.1e}'.format(Ra))
    print('q1 = {0:.3f}; q2 = {1:.3f}'.format(q1, q2))
    print('q3 = {0:.3f}; q4 = {1:.3f}'.format(q3, q4))
```

```python
# Let's add a test to ensure things are working as expected
if case == "a":
    if not np.isclose(q1,8.020,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected q1.")
    if not np.isclose(q2,0.589,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected q2.")
```

**Save time and rms values**

The following command uses the ``numpy`` package save to text file function to output all $v_{RMS}$ values as a function of time. This is particularly useful if you have run the simulation from the perturbed initial condition rather than the saved data file, as you can see the system coming to steady state.

The format for this text file is:

    timeVal[0], vrmsVal[0]
    timeVal[1], vrmsVal[1]
    ...
    timeVal[N], vrmsVal[N]

```python
if(uw.mpi.rank==0):
    np.savetxt(outputPath+'vrms.txt', np.c_[timeVal, vrmsVal], header="Time, VRMS" )
```

**Calculate stress values for benchmark comparison**

Determine stress field for whole box in dimensionless units (King 2009)
\begin{equation}
\tau_{ij} = \eta \frac{1}{2} \left[ \frac{\partial v_j}{\partial x_i} + \frac{\partial v_i}{\partial x_j}\right]
\end{equation}
which for vertical normal stress becomes
\begin{equation}
\tau_{zz} = \eta \frac{1}{2} \left[ \frac{\partial v_z}{\partial z} + \frac{\partial v_z}{\partial z}\right] = \eta \frac{\partial v_z}{\partial z}
\end{equation}
which is implemented for the whole box in the functions defined below.

```python
# get topography from non-partitioned stress tensor
if(uw.mpi.rank==0):
    stresstensorFn = 2.* stokes.fn_viscosity*fn.tensor.symmetric( velocityField0.fn_gradient ) - (1.,1.,0.)*pressureField0
    verticalStressFn  = stresstensorFn[1]
    stress_zz_top     = -verticalStressFn.evaluate(mesh0.specialSets["MaxJ_VertexSet"])

# subtract the average value for benchmark.
    mean_sigma_zz_top = np.mean(stress_zz_top)
    sigma_zz_top      = stress_zz_top - mean_sigma_zz_top
```

Dimensionalise the stress from the vertical normal stress at the top of the box (King 2009)

$$
    \sigma_{t} = \frac{\eta_0 \kappa}{\rho g h^2}\tau _{zz} \left( x, z=h\right)
$$

where all constants have been defined above. Finally calculate the topography, defined using $h = \sigma_{top} / (\rho g)$.

```python
# Set parameters in SI units
if(uw.mpi.rank==0):
    grav = 10        # m.s^-2
    height = 1.e6    # m
    rho  = 4.0e3     # g.m^-3
    kappa  = 1.0e-6  # m^2.s^-1

# dimensionalise
    dim_sigma_zz_top  = (eta0 * kappa / (height*height)) * sigma_zz_top

# find topography in [m]
    topography        = dim_sigma_zz_top / (rho * grav)
```

**Calculate x-coordinate at zero stress**

Calculate the zero point for the stress along the x-axis at the top of the box using the **interpolation function** from ``numpy``. Note that ``numpy`` requires that the first array input for ``np.interp`` must be increasing, so the negative of the topography is used.

```python
if(uw.mpi.rank==0):
    xCoordFn = fn.input()[0]
    x = xCoordFn.evaluate(mesh0.specialSets["MinJ_VertexSet"])
    xIntercept = np.interp(0.0,-1.0*topography[:, 0],x[:, 0])
```

**Topography comparison**

Topography of the top boundary calculated in the left and right corners as given in Table 9 of Blankenbach et al 1989.

| $Ra$          |    $\xi_1$  | $\xi_2$  |  $x$ ($\xi = 0$) |
| ------------- |:-----------:|:--------:|:--------------:|
| 10$^4$  | 2254.02   | -2903.23  | 0.539372          |
| 10$^5$  | 1460.99   | -2004.20  | 0.529330          |
| 10$^6$  | 931.96   | -1283.80  | 0.506490          |

```python
if(uw.mpi.rank==0):
    e1 = float(topography[0])
    e2 = float(topography[len(topography)-1])
    print('Rayleigh number = {0:.1e}'.format(Ra))
    print('Topography[x=0],[x=max] = {0:.2f}, {1:.2f}'.format(e1, e2))
    print('x(topo=0) = {0:.6f}'.format(xIntercept))
    # output a summary file with benchmark values (useful for parallel runs)
    np.savetxt(outputPath+'summary.txt', [Ra, e1, e2, xIntercept, q1, q2, q3, q4])
```

```python
# Let's add a test to ensure things are working as expected
if case == "a":
    if not np.isclose(e1,2254.02,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected xi1.")
    if not np.isclose(e2,-2903.23,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected xi2.")
```


# Intro to Deep Learning
We're going to continue working with CAP imagery for the second half of this week. Recall the two main guiding questions for this week:
- _What_ is in an image (e.g. debris, buildings, etc.)?
- _Where_ are these things located _in 3D space_ ?

## Motivation

We've already seen how structure from motion can begin to answer the second of those questions, at least in a coarse manner. The first question is more difficult than the second for one crucial reason: it (so far) can not be answered *solely* using image data. Recall that structure from motion was able to leverage image metadata, features and geometric constraints without having to insert any "outside" data. When detecting the components of an image, there is a certain amount of subjectivity that so far requires human input. For example, if you want to detect flooding in an image, there need to be some set of rules that the software can determine "if (rules are satisfied), then (flooding = True)".

How can you start to tackle this problem? We're going to make the problem a bit simpler by turning it into a classification problem rather than a localization problem. That is, instead of finding where flooding is in an image, we'll find whether there is flooding at all. With that in mind, let's first try a naive approach and see if we can simply enumerate the rules. Let's look at some flooding images:

So what do we see? It looks like flooding is this murky, brown color that covers most of the image. So let's make a rule: if some percentage of the image is this brownish color (you can think of detecting this by creating some sort of index, like you did with the satellite imagery), there is flooding in the image. This might work for the two images we saw, but the real test is seeing if it works with other images. So let's look at another one:

This image has a large percentage of it covered by brownish looking water. However, it's clearly not flooding, but rather just a lake. No problem, let's just make another rule: if some percentage of the image is this brownish color *and* there are also lots of buildings (let's also make an index for that), there is flooding in the image. Let's see if this works:

Again, this is just some lake (the same lake as before, in fact). At this point, we might be thinking of shifting our strategy a bit. Clearly having someone sit down and enumerate all of the rules is impractical in this case\*. There is simply too much variability in the images to come up with a set of rules that is remotely generalizable.

This is one of the major motivations of machine learning. The idea is that, rather than enumerate the rules that determine flooding, we're going to just enumerate the outcomes (e.g. whether there is flooding or not) and develop algorithms for the computer to *learn* what the rules are. There is a rich literature (and a host of undergraduate and graduate courses) on how these algorithms are designed, but on these next couple of days we will instead focus on how to interpret and implement them.

\* This isn't to say that enumerating the rules isn't practical in *every* case. It really depends on the characteristics of the data.

## A simple case study

We're going to start with a simple dataset. We will use the  Breast Cancer Wisconsin (Diagnostic) Data Set (https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)), a dataset that shows whether a tumor is benign or malign and various other features.

```python
# If the cell below returns an error related to "as_frame" not working,
# you need to update scikit-learn. To do this, uncomment the next
# line and restart the kernel
!pip install scikit-learn==0.23.1
```

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.datasets import load_breast_cancer

# read the dataframe
df = load_breast_cancer(return_X_y=False, as_frame=True)["data"]
df["target"] = load_breast_cancer(return_X_y=False, as_frame=True)["target"] # benign is 1, malign is 0
df
```

```python
# for now we're just going to focus on mean radius and mean texture.
# we'll plot one against the other, and color code depending on survival
df_b = df[df["target"] == 1]
df_m = df[df["target"] == 0]

# plotting
plt.plot(df_b["mean radius"], df_b["mean texture"], "ro")
plt.plot(df_m["mean radius"], df_m["mean texture"], "bo")
plt.xlabel("mean radius")
plt.ylabel("mean texture")
plt.show()
```

So what do we see? Clearly there are factors beyond the ones plotted that explain whether a tumor is benign. However, it's not difficult to see a pattern here. The further up and to the right you are, the less likely you are to survive. So as a first attempt, we will have the software learn a *linear classifier*. This classifier will be a line that will classify you as benign if you are below the line, and malign otherwise.

```python
from sklearn import svm
from sklearn.model_selection import train_test_split
try:
    from mlxtend.plotting import plot_decision_regions
except:
    !pip install mlxtend
    from mlxtend.plotting import plot_decision_regions

# setting the features and the targets
X = df[["mean radius", "mean texture"]]
y = df["target"]

# splitting training and testing set 80/20
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Initialize SVM classifier
classifier = svm.SVC(kernel='linear')

# Fit data
classifier = classifier.fit(X_train, y_train)

# Printing parameters
coef = classifier.coef_
print(coef)

# Plot decision boundary
plot_decision_regions(X_train.values, y_train.values.astype(np.integer), clf=classifier, legend=2)
plt.xlabel("mean radius")
plt.ylabel("mean texture")
plt.show()
```

There is a lot going on here, so let's try to break it down. Recall that we are trying to determine whether a patient survives using just the (normalized) age and detected nodes. These are called the explanatory variables or *features*, and we refer to it as $X$. The variable we are trying to predict is the prediction variable or *target*, and we refer to it as $y$.

The underlying notion behind machine learning is that patterns that hold in a subset of data will hold in the population as a whole. So in a sense, we care about the performance of the classifier on data that the algorithm has *not* seen by learning patterns in that data that we have seen. To this end, one of the major paradigms in machine learning is to separate our data into a *training set* and a *testing set*. The training set is a subset of the given data that we will actually provide to the algorithm to learn, while the testing set will be used solely to report the performance on data that the algorithm has not seen. In our case, we set 80% of the data as training data and 20% as testing data.

There are numerous algorithms that can train on data. One of the more popular ones (especially before neural networks were introduced) are called *support vector machines* (SVM). We will not discuss how it actually does the learning, but suffice to say it takes in the training data and finds the line that minimizes the classification error. This line is called the *decision boundary* and it is defined by the following equation:

\begin{equation}
C_1(mean\ radius) + C_2(mean\ texture) = 0
\end{equation}

Where $C_1$ and $C_2$ are the coefficients of the linear classifier. So really, the function of the SVM is to find these two coefficients. The algorithm then does classification as follows:

\begin{equation}
y_{pred} =
\begin{cases}
    1               & \text{if } C_1(mean\ radius) + C_2(mean\ texture)\geq 0\\
    0,              & \text{otherwise}
\end{cases}
\end{equation}

### Exercise
- Look at the first 5 samples in the testing data set. Calculate the predicted target of each of the 5 samples?
- Look at the corresponding targets. Are the predictions accurate? What proportion of the predicted values are correctly predicted?
- Which of the two coefficients is most important? Why?

```python

```

How good is our classifier? We can look at the performance of our classifier on the testing set to find out. Of course, an important metric to look out for is the accuracy:

\begin{equation}
\text{Accuracy} = \frac{\#\ (y_{true} == y_{pred})}{\# \ \text{samples}}
\end{equation}

However, for most applications it is also worth understanding what your *false positive* and *false negative* rate is. A false positive is a sample that is classified as a 1 but is in reality a 0, and a false negative is the opposite\*. The false positive rate is defined as $\frac{\# \text{False positives}}{\# \ \text{False positives + True negatives}}$, while the false negative rate is defined as $\frac{\# \text{False negatives}}{\# \ \text{False negatives + True positives}}$. **Why would this be a metric we care about, especially in this case?**

    ANSWER
    Imagine your tumor is predicted as malignant but is in fact benign. There is certainly now a cost you incur in further tests or surgeries, but ultimately it is a manageable cost. Now imagine if you predicted a tumor as benign even thoguh it is malignant. Here, the mistake of the classifier could certainly cost you your life.

It turns out there is a useful tool called a *confusion matrix* that handily lays it out for you.

\* It's worth putting some effort into what we call positives and negatives. Here, we use the term positive to mean 1 (benign) and negative to mean 0 (malign). However, anyone that has had a test of any sort knows that positive usually means that you have the disease (in our case, malign tumor). A lot of data science is communicating results, so something worth thinking about.

```python
from sklearn.metrics import plot_confusion_matrix

predictions = classifier.predict(X_test)
accuracy = np.sum(predictions == y_test)/np.size(predictions)
print(accuracy)

matrix = plot_confusion_matrix(classifier, X_test, y_test,
                                 cmap=plt.cm.Blues,
                                 normalize='true')
plt.title('Confusion matrix for our classifier')
plt.show(matrix)
plt.show()
```

As a final note, we used two variables (radius and texture) because it made it possible to visualize. However, there is nothing preventing us from using the rest of the data set columns as part of our features. While visualization in more than 2D is much more difficult, the underlying principles are exactly the same.

### Exercise
Explore this link which contains a number of classification datasets: https://archive.ics.uci.edu/ml/datasets.php?format=&task=cla&att=&area=&numAtt=&numIns=&type=&sort=nameUp&view=table. Using an SVM, attempt to predict the class of each sample. Consider the following:
- While multiclass classification is possible, limit yourself thus far to datasets that have one class. Make sure that y is set up with two distinct numbers for the classes (e.g. 0 and 1)
- Make sure that your dataset does not have missing values. If it does, handle them appropriately (e.g. by substituting it with 0 or with the mean of the column).

How does your classifier perform? Report the accuracy and confusion matrix. What could be done to improve the performance?

```python

```

## Deep learning
So far, we've worked exclusively with linear decision boundaries. However, usually data is much more complicated (as you should have seen from the exercise). Therefore, we might want to look for something that has higher predictive power. One of the most popular tools in machine learning has been the *neural network*. A neural network is (loosely) inspired by the way neurons are connected and operate. Let's first look at the simplest possible neural network and work our way up:

Neural networks are made up of *layers*. In the image above, the neural network has two layers. The first layer is called the *input layer*, and the last layer is called the *output layer*. Each layer is composed of *neurons*. In the input layer, each neuron is one of the features from the training data. For example, the first neuron might be the mean radius, and the second neuron might be the mean texture. We're going to refer to each neuron by $x_{l, n}$, where $l$ is the number of the layer and $n$ is the number of the neuron within the layer. Using this notation, mean radius = $x_{1, 1}$.

Each neuron in the input layer is connected to *every* neuron in the following layer by a *weight*. We will denote each weight by $w_{l, i, o}$, where $l$ is the layer at the tail of the arrow, $i$ is the number of the neuron at the tail of the arrow and $o$ is the number of the neuron at the head of the arrow. In the figure above, the color of the arrow corresponds to the value of the weight (blue means negative, red means positive). Every neuron in subsequent layers is equal to the sum of the products of the neurons and the weight that connects them. In this case, the neuron in the output layer is defined as:

\begin{equation}
x_{2, 1} = x_{1, 1}w_{1, 1, 1} + x_{1, 2}w_{1, 2, 1}
\end{equation}

Once you have the value of the output layer, you can obtain the predicted classification by applying some function to $x_{2, 1}$. An example of such a function would be similar to what we did previously:

\begin{equation}
y_{pred} =
\begin{cases}
    1               & \text{if } x_{2, 1}\geq 0\\
    0,              & \text{otherwise}
\end{cases}
\end{equation}

So training a neural network is simply finding the weights $w$ that minimize the error. If you are keen-eyed though, you might have realized that the example above is identical to what we were doing before. Therefore, we still have not arrived at a more powerful classifier. Let's try a more complicated neural network.

This network is similar to the previous one, but now we have an additional layer in between the input and output layers. All layers that are between input and output are called *hidden layers*, though functionally they behave in exactly the same way. As a quick concept question, **what is the value of $x_{2, 3}$? What about $x_{3, 1}$?**

    ANSWER
    $x_{2, 3} = x_{1, 1}w_{1, 1, 3} + x_{1, 2}w_{1, 2, 3}$ ,
    $x_{3, 1} = x_{2, 1}w_{2, 1, 1} + x_{2, 2}w_{2, 2, 1} + x_{2, 3}w_{2, 3, 1} + x_{2, 4}w_{2, 4, 1} + x_{2, 5}w_{2, 5, 1}$

Why are we introducing a hidden layer? Notice that each of the neurons in the hidden layer are essentially acting as linear classifiers. By having a weighted sum of these neurons in the output layer, we are arriving at a function that, by virtue of being a sum of lines is not itself linear. One of the biggest results in machine learning is that, as you increase the number of neurons in the hidden layer, you can approximate *any* function, meaning that you can always make a more powerful classifier by just adding more neurons. However, usually this is not computationally efficient. Rather, most of the time it makes more sense to introduce more hidden layers as opposed to more neurons, as shown in the image below:

The added "depth" as a result of more layers is where the "deep" in "deep learning" comes. As computational resources have increased, scientists have increasingly resorted to adding more and more layers to their neural networks.

### Exercise
Let's start thinking about the LADI dataset. We want to use deep learning to detect whether there is flooding in the image.
- Using the paradigm above, what would be the features of the dataset? What are the targets?
- Propose a way that you would input the features into a neural network?
- What problems might you run into if you directly put an image into the neural network above?

```python

```

## Convolution
Convolution is a fairly old technique in image processing. Convolution works by multiplying a matrix (called a *kernel*) with portions of the image and adding up the result. An animation of how this works is shown below. The result of applying different kernels is also shown:

As you can see, different kernels can pick up different features depending on the specific coefficients of the kernel. In the late 1980s, scientists first thought to use neural networks to learn the kernel values themselves, which created the foundation for the *convolutional* neural network (CNN). These complement the hidden layers from the previous architecture with new convolutional layers, which perform convolution on the previous layer. In so doing, these networks conserve the relationship between nearby pixels and perform very well in image classification.

## The LADI dataset
As we already saw, the LADI dataset contains CAP images as well as crowdsourced labels for a number of different categories. Today we're going to start the multi-day exercise of applying deep learning to the LADI dataset in order to do classification of flooding. This section borrows heavily from the following tutorial: https://github.com/LADI-Dataset/ladi-tutorial

### Data reading and cleaning
We first have to go through some work to make sure that we only have the components of the dataset that we want.

```python
# read the tsv file with the labels
file = pd.read_csv("http://ladi.s3-us-west-2.amazonaws.com/Labels/ladi_aggregated_responses_url.tsv",delimiter='\t',header='infer')
file
```

```python
# Strip off bracket and comma from the Answer catagory
file["Answer"] = file["Answer"].str.strip('[|]')
file["Answer"] = file["Answer"].str.split(",",expand = True)

# Extract labels with damage and infrastructure categories
label_damage_infra = file[file['Answer'].str.contains('damage|infrastructure',na=False,case=False)]

#Filter out infrastructure label with label 'none'
label_clean = label_damage_infra[~label_damage_infra['Answer'].str.contains('none',na=False,case=False)]

# Extract data with label does contain 'flood'
label_flood = label_clean[label_clean['Answer'].str.contains('flood',na=False,case=False)]

# Extract url data with the label does contain 'flood'
im_flood_lst = label_flood['url'].unique().tolist()

# Extract url data with the label does not contain 'flood'
label_notflood = label_damage_infra[~label_damage_infra['url'].isin(im_flood_lst)]
im_not_flood_lst = label_notflood['url'].unique().tolist()
```

Now that we have the images we're interested in, we're going to generate the true/false labels

```python
# Load ladi_images_metadata.csv
metadata = pd.read_csv('http://ladi.s3-us-west-2.amazonaws.com/Labels/ladi_images_metadata.csv')

# Generate flood and non-flood metadata
flood_metadata = metadata[metadata['url'].isin(im_flood_lst)]
not_flood_metadata = metadata[metadata['url'].isin(im_not_flood_lst)]

# Generate url and s3_path features into list
flood_meta_lst = flood_metadata['url'].tolist()
flood_meta_s3_lst = flood_metadata['s3_path'].tolist()

not_flood_meta_lst = not_flood_metadata['url'].tolist()
not_flood_meta_s3_lst = not_flood_metadata['s3_path'].tolist()

# Check how many images do not have metadata but have human labels
human_label_only = list(set(im_flood_lst) - set(flood_meta_lst))
print(len(human_label_only))
human_label_non_flood = list(set(im_not_flood_lst) - set(not_flood_meta_lst))
print(len(human_label_non_flood))
```

For the purposes of this exercise, we're going to take just a sample of the overall imagery. We're going to take 100 images that are labeled as flood and 100 that are not:

```python
from random import sample
# sampling
flood_tiny_lst = sample(flood_meta_s3_lst, 100)
not_flood_tiny_lst = sample(not_flood_meta_s3_lst, 100)
flood_tiny_metadata = metadata[metadata['s3_path'].isin(flood_tiny_lst+not_flood_tiny_lst)]

# creating the new datasets
flood_data = []
for path in flood_tiny_lst:
    data_lst = []
    data_lst.append(path)
    data_lst.append(True)
    flood_data.append(data_lst)

not_flood_data = []
for path in not_flood_tiny_lst:
    data_lst = []
    data_lst.append(path)
    data_lst.append(False)
    not_flood_data.append(data_lst)

label_data = flood_data+not_flood_data
label_df = pd.DataFrame(label_data, columns = ['s3_path', 'label'])

flood_tiny_metadata.to_csv('flood_tiny_metadata.csv')
label_df.to_csv('flood_tiny_label.csv')
```

### Exercise
Take a look at the LADI csv files. Find some classification task that you find meaningful (e.g. detecting a bridge in an image). Create a csv file like the ones above for the metadata and the label. You might have to do some additional steps in data cleaning, depending on your choice

```python

```


```python
import sys
sys.path.append('../')
sys.path.append('../../')

%matplotlib inline

from errorpro.default import *
p = default_project
```

```python
load("teil1.dat")
```

```python
fit("a4*p**4 + a3*p**3 + a2*p**2 + a1*p + a0", ["p","B"], ["a0","a1","a2","a3","a4"])
```

Displaying: $a_{0}$, $a_{1}$, $a_{2}$, $a_{3}$, $a_{4}$
<button onclick='var e = document.getElementById("d1").style;if (e.display=="none"){e.display="block";}else {e.display="none";}'>Data</button><button onclick='var e = document.getElementById("d2").style;if (e.display=="none"){e.display="block";}else {e.display="none";}'>LaTeX</button>

		$a_{0} \; \mathrm{\left[T\right]}$
		$a_{1} \; \mathrm{\left[\frac{Pa}{A}\right]}$
		$a_{2} \; \mathrm{\left[\frac{Pa}{A m}\right]}$
		$a_{3} \; \mathrm{\left[\frac{Pa}{A m^{2}}\right]}$
		$a_{4} \; \mathrm{\left[\frac{Pa}{A m^{3}}\right]}$

		$1.198 \pm 0.017$
		$-22.1 \pm 1.8$
		$160 \pm 70$
		$(5 \pm 10) \times 10^{2}$
		$(-10 \pm 6) \times 10^{3}$

 \begin{table}[H]
\centering
	\begin{tabular}{|c|c|c|c|c|}
	\hline
	$a_{0} \; \mathrm{\left[T\right]}$ & $a_{1} \; \mathrm{\left[\frac{Pa}{A}\right]}$ & $a_{2} \; \mathrm{\left[\frac{Pa}{A m}\right]}$ & $a_{3} \; \mathrm{\left[\frac{Pa}{A m^{2}}\right]}$ & $a_{4} \; \mathrm{\left[\frac{Pa}{A m^{3}}\right]}$\\ \hline
	$1.198 \pm 0.017$ & $-22.1 \pm 1.8$ & $160 \pm 70$ & $(5 \pm 10) \times 10^{2}$ & $(-10 \pm 6) \times 10^{3}$\\ \hline
	\end{tabular}
\end{table}

```python
# differentiate fit function
from sympy import diff
from errorpro.quantities import parse_expr
fit_function = parse_expr("a4*p**4 + a2*p**2 + a1*p + a0",p.data)
ableitung = diff(fit_function, p.data["p"])
BdBdh = fit_function*ableitung

# save values of B*dB/dh at certain positions
Bi = BdBdh.subs(p["p"],p.data["p_Bi_m"])
assign("BdBdh_Bi", value=Bi, unit="T**2/m")
Mn = BdBdh.subs(p.data["p"],p.data["p_Mn_m"])
assign("BdBdh_Mn", value=Mn, unit="T**2/m")
Ta = BdBdh.subs(p.data["p"],p.data["p_Ta_m"])
assign("BdBdh_Ta", value=Ta, unit="T**2/m")
```

```python
%%calc
chi_Bi = F_Bi*mu0/(m_Bi/rho_Bi)/BdBdh_Bi
chi_Mn = F_Mn*mu0/(m_Mn/rho_Mn)/BdBdh_Mn
chi_Ta = F_Ta*mu0/(m_Ta/rho_Ta)/BdBdh_Ta
```

```python
table("chi_Bi","chi_Mn","chi_Ta")
```

Displaying: $\chi_{Bi}$, $\chi_{Mn}$, $\chi_{Ta}$
<button onclick='var e = document.getElementById("d3").style;if (e.display=="none"){e.display="block";}else {e.display="none";}'>Data</button><button onclick='var e = document.getElementById("d4").style;if (e.display=="none"){e.display="block";}else {e.display="none";}'>LaTeX</button>

		$\chi_{Bi} \; \mathrm{\left[1\right]}$
		$\chi_{Mn} \; \mathrm{\left[1\right]}$
		$\chi_{Ta} \; \mathrm{\left[1\right]}$

		$(-1.2 \pm 0.4) \times 10^{-4}$
		$(5.9 \pm 1.6) \times 10^{-4}$
		$(2.4 \pm 0.8) \times 10^{-4}$

 \begin{table}[H]
\centering
	\begin{tabular}{|c|c|c|}
	\hline
	$\chi_{Bi} \; \mathrm{\left[1\right]}$ & $\chi_{Mn} \; \mathrm{\left[1\right]}$ & $\chi_{Ta} \; \mathrm{\left[1\right]}$\\ \hline
	$(-1.2 \pm 0.4) \times 10^{-4}$ & $(5.9 \pm 1.6) \times 10^{-4}$ & $(2.4 \pm 0.8) \times 10^{-4}$\\ \hline
	\end{tabular}
\end{table}

```python
load("teil2.dat")
```

```python

```


#**Chapter 4 – Training Models**

     Open the original notebook

Let's begin like in the last notebook: importing a few common modules, ensuring MatplotLib plots figures inline and preparing a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so once again we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20.

You don't need to worry about understanding everything that is written in this section.

```python
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Is this notebook running on Colab or Kaggle?
IS_COLAB = "google.colab" in sys.modules

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

# Common imports
import numpy as np
import os

# To make this notebook's output stable across runs
rnd_seed = 42
rnd_gen = np.random.default_rng(rnd_seed)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "classification"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)
```

In this notebook we will be working with the [*Iris Flower Dataset*](https://en.wikipedia.org/wiki/Iris_flower_data_set), in which the length and width of both the sepals and petals of three types of Iris flowes were recorded. For reference, these are pictures of the three flowers:

 In order: Iris Setosa,  Iris Versicolor, and Iris Virginica

Photo Credits:[Kosaciec szczecinkowaty Iris setosa](https://en.wikipedia.org/wiki/File:Kosaciec_szczecinkowaty_Iris_setosa.jpg) by [Radomil Binek](https://commons.wikimedia.org/wiki/User:Radomil) licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en); [Blue flag flower close-up (Iris versicolor)](https://en.wikipedia.org/wiki/File:Iris_versicolor_3.jpg)by Danielle Langlois licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en); [image of Iris virginica shrevei](https://en.wikipedia.org/wiki/File:Iris_virginica.jpg) by [Frank Mayfield](https://www.flickr.com/photos/33397993@N05) licensed under [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/deed.en).

As you can imagine, this dataset is normally used to train *multiclass*/*multinomial* classification algorithms and not *binary* classification algorithms, since there *are* more than 2 classes.

"*Three classes, even!*" - an observant TA

For this exercise, however, we will be implemented the binary classification algorithm referred to as the *logistic regression* algorithm (also called logit regression).

```python
# Let's load the Iris Dataset
from sklearn import datasets
iris = datasets.load_iris()

# Print out some information about the data
print(f'Keys in Iris dictionary: \n{list(iris.keys())}\n\n')
print(iris.DESCR)

# And load the petal lengths and widths as our input data
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]

print(iris['data_module'])

# The target data labels Setosa as 0, Versicolor as 1, and Virginica as 2. For
# this exercise we will be using only the Versicolor and Virgina sets.
bin_indices = np.logical_or(y==1,y==2)
bin_X = X[bin_indices]
bin_y = (y[bin_indices]==2).astype(np.uint8) # convert to binary

print(bin_X)
```

    Keys in Iris dictionary:
    ['data', 'target', 'frame', 'target_names', 'DESCR', 'feature_names', 'filename', 'data_module']

    .. _iris_dataset:

    Iris plants dataset
    --------------------

    **Data Set Characteristics:**

        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica

        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
    from Fisher's paper. Note that it's the same as in R, but not as in the UCI
    Machine Learning Repository, which has two wrong data points.

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    .. topic:: References

       - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...
    sklearn.datasets.data
    [[4.7 1.4]
     [4.5 1.5]
     [4.9 1.5]
     [4.  1.3]
     [4.6 1.5]
     [4.5 1.3]
     [4.7 1.6]
     [3.3 1. ]
     [4.6 1.3]
     [3.9 1.4]
     [3.5 1. ]
     [4.2 1.5]
     [4.  1. ]
     [4.7 1.4]
     [3.6 1.3]
     [4.4 1.4]
     [4.5 1.5]
     [4.1 1. ]
     [4.5 1.5]
     [3.9 1.1]
     [4.8 1.8]
     [4.  1.3]
     [4.9 1.5]
     [4.7 1.2]
     [4.3 1.3]
     [4.4 1.4]
     [4.8 1.4]
     [5.  1.7]
     [4.5 1.5]
     [3.5 1. ]
     [3.8 1.1]
     [3.7 1. ]
     [3.9 1.2]
     [5.1 1.6]
     [4.5 1.5]
     [4.5 1.6]
     [4.7 1.5]
     [4.4 1.3]
     [4.1 1.3]
     [4.  1.3]
     [4.4 1.2]
     [4.6 1.4]
     [4.  1.2]
     [3.3 1. ]
     [4.2 1.3]
     [4.2 1.2]
     [4.2 1.3]
     [4.3 1.3]
     [3.  1.1]
     [4.1 1.3]
     [6.  2.5]
     [5.1 1.9]
     [5.9 2.1]
     [5.6 1.8]
     [5.8 2.2]
     [6.6 2.1]
     [4.5 1.7]
     [6.3 1.8]
     [5.8 1.8]
     [6.1 2.5]
     [5.1 2. ]
     [5.3 1.9]
     [5.5 2.1]
     [5.  2. ]
     [5.1 2.4]
     [5.3 2.3]
     [5.5 1.8]
     [6.7 2.2]
     [6.9 2.3]
     [5.  1.5]
     [5.7 2.3]
     [4.9 2. ]
     [6.7 2. ]
     [4.9 1.8]
     [5.7 2.1]
     [6.  1.8]
     [4.8 1.8]
     [4.9 1.8]
     [5.6 2.1]
     [5.8 1.6]
     [6.1 1.9]
     [6.4 2. ]
     [5.6 2.2]
     [5.1 1.5]
     [5.6 1.4]
     [6.1 2.3]
     [5.6 2.4]
     [5.5 1.8]
     [4.8 1.8]
     [5.4 2.1]
     [5.6 2.4]
     [5.1 2.3]
     [5.1 1.9]
     [5.9 2.3]
     [5.7 2.5]
     [5.2 2.3]
     [5.  1.9]
     [5.2 2. ]
     [5.4 2.3]
     [5.1 1.8]]

We now have a set of binary classification data we can use to train an algorithm.

As we saw during our reading, we need to define three things in order to train our algorithm: the type of algorithm we will train, the cost function (which will tell us how close our prediction is to the truth), and a method for updating the parameters in our model according to the value of the cost function (e.g., the gradient descent method).

Let's begin by defining the type of algorithm we will use. We will train a logistic regression model to differentiate between two classes. A reminder of how the logistic regression algorithm works is given below.

The logistic regression algorithm will thus take an input $t$ that is a linear combination of the features:

 $t_{\small{n}} = \beta_{\small{0}} + \beta_{\small{1}} \cdot X_{1,n} + \beta_{\small{2}} \cdot X_{2,n}$

where
* $n$ is the ID of the sample
* $X_{\small{0}}$ represents the petal length
* $X_{\small{1}}$ represents the petal width

This input is then fed into the logistic function, $\sigma$:
\begin{align}
\sigma: t\mapsto \dfrac{1}{1+e^ {-t}}
\end{align}

Let's plot it below to remember the shape of the function

```python
t = np.arange(-4,4,.1)
def logistic(in_val):
    # Return the value of the logistic function
    return 1/(1 + np.exp(- in_val))

fig, ax = plt.subplots()

ax.axvline(0, c='black', alpha=1)
ax.axhline(0, c='black', alpha=1)

[ax.axhline(y_val, c='black', alpha=0.5, linestyle='dotted') for y_val in (0.5,1)]

plt.autoscale(axis='x', tight=True)

ax.plot(t, logistic(t));
ax.set_xlabel('$t$')
ax.set_ylabel('$\\sigma\\  \\left(t\\right)$')
fig.tight_layout()
```

With the logistic function, we define inputs resulting in $\sigma\geq.5$ as belonging to the ***one*** class, and any value below that is considered to belong to the ***zero*** class.

We now have a function which lets us map the value of the petal length and width to the class to which the observation belongs (i.e., whether the length and width correspond to Iris Versicolor or Iris Virginica). However, there is a parameter vector **$\theta$** with a number of parameters that we do not have a value for:
 $\theta = [ \beta_{\small{0}}, \beta_{\small{1}}$, $\beta_{\small{2}} ]$

**Q1) Set up an array of random numbers between 0 and 1 representing the $\theta$ vector.**

Hint:  Use `rnd_gen`! If you're not sure how to use it, consult the `default_rng` documentation [at this link](https://numpy.org/doc/stable/reference/random/generator.html). For instance, you may use the `random` method of `rnd_gen`.

```python
# Write your code here
theta = np.random.randint(2,size=(3,1))
print(theta)
```

    [[0]
     [1]
     [1]]

In order to determine whether a set of $\beta$ values is better than the other, we need to quantify well the values are able to predict the class. This is where the cost function comes in.

The cost function, $c$, will return a value close to zero when the prediction, $\hat{p}$, is correct and a large value when it is wrong. In a binary classification problem, we can use the log loss function. For a single prediction and truth value, it is given by:
\begin{align}
        \text{c}(\hat{p},y) = \left\{
        \begin{array}{cl}
        -\log(\hat{p})& \text{if}\; y=1\\
        -\log(1-\hat{p}) & \text{if}\; y=0
        \end{array}
        \right.
    \end{align}

However, we want to apply the cost function to an n-dimensional set of predictions and truth values. Thankfully, we can find the average value of the log loss function $J$ for an an-dimensional set of $\hat{y}$ & $y$ as follows:

\begin{align}
        \text{J}(\mathbf{\hat{p}},y) = - \dfrac{1}{n} \sum_{i=1}^{n}
        \left[ y_i\cdot \log\left( \hat{p}_i \right) \right] +
        \left[ \left( 1 - y_i \right) \cdot \log\left( 1-\hat{p}_i \right) \right]
    \end{align}

We now have a formula that can be used to calculate the average cost over the training set of data.

**Q2) Define a log_loss function that takes in an arbitrarily large set of prediction and truths**

Hint 1: You need to encode the function $J$ above, for which Numpy's functions may be quite convenient (e.g., [`log`](https://numpy.org/doc/stable/reference/generated/numpy.log.html), [`mean`](https://numpy.org/doc/stable/reference/generated/numpy.mean.html), etc.)

Hint 2: Asserting the dimensions of the vector is a good way to check that your function is working correctly. [Here's a tutorial on how to use `assert`](https://swcarpentry.github.io/python-novice-inflammation/10-defensive/index.html#assertions). For instance, to assert that two vectors `X` and `Y` have the same dimension, you may use:
```
assert X.shape==Y.shape
```

Now let's code 💻

```python
def log_loss(p_hat, y, epsilon=1e-7):
  # Write your code here.
  # We can also run into problems if p_hat = 0, so add an _epsilon_ term
  # when evaluating log(p_hat).
  assert y.shape[0]==p_hat.shape[0], "Vectors must have the same lenth"
  log_p_hat = epsilon + np.log(p_hat)
  n = len(p_hat)
  j1 = np.sum(y * log_p_hat)
  j2 = np.sum((1-y) * np.log(1 - p_hat))
  J = - 1/n * (j1+j2)

  # After calculating J, assert that J has the same shape as p_hat and y
  # I DONT UNDERSTAND, J IS A SCALAR, IT CANNOT HAVE THE SAME SHHAPE AS p_hat and y
  # assert J.shape==y.shape
  # assert J.shape==p_hat.shape
  return J
```

We now have a way of quantifying how good our predictions are. The final thing needed for us to train our algorithm is figuring out a way to update the parameters in a way that improves the average quality of our predictions.

**Warning**: we'll go into a bit of math below

Let's look at the change in a single parameter within $\theta$: $\beta_1$ (given $X_{1,i} = X_1$, $\;\hat{p}_{i} = \hat{p}$, $\;y_{i} = y$). If we want to know what the effect of changing the value of $\beta_1$ will have on the log loss function we can find this with the partial derivative:
$
        \dfrac{\partial J}{\partial \beta_1}
$

This may not seem very helpful by itself - after all, $\beta_1$ isn't even in the expression of $J$. But if we use the chain rule, we can rewrite the expression as:

        $\dfrac{\partial J}{\partial \hat{p}} \cdot
        \dfrac{\partial \hat{p}}{\partial \theta} \cdot
        \dfrac{\partial \theta}{\partial \beta_1}$

We'll spare you the math (feel free to verify it youself, however!):

$\dfrac{\partial J}{\partial \hat{p}} =  \dfrac{\hat{p} - y}{\hat{p}(1-\hat{p})}, \quad
        \dfrac{\partial \hat{p}}{\partial \theta} = \hat{p} (1-\hat{p}), \quad
        \dfrac{\partial \theta}{\partial \beta_1} = X_1 $

and thus
$
        \dfrac{\partial J}{\partial \beta_1} = (\hat{p} - y) \cdot X_1
$

We can calculate the partial derivative for each parameter in $\theta$ which, as you may have realized, is simply the $\theta$ gradient of $J$: $\nabla_{\theta}(J)$

With all of this information, we can now write $\nabla_{\theta} J$ in terms of the error, the feature vector, and the number of samples we're training on!

$\nabla_{\mathbf{\theta}^{(k)}} \, J(\mathbf{\theta^{(k)}}) = \dfrac{1}{n} \sum\limits_{i=1}^{n}{ \left ( \hat{p}^{(k)}_{i} - y_{i} \right ) \mathbf{X}_{i}}$

Note that here $k$ represents the iteration of the parameters we are currently on.

We now have a gradient we can calculate and use in the batch gradient descent method! The updated parameters will thus be:

\begin{align}
{\mathbf{\theta}^{(k+1)}} = {\mathbf{\theta}^{(k)}} - \eta\,\nabla_{\theta^{(k)}}J(\theta^{(k)})
\end{align}

Where $\eta$ is the learning rate parameter. It's also worth pointing out that $\;\hat{p}^{(k)}_i = \sigma\left(\theta^{(k)}, X_i\right) $

In order to easily calculate the input to the logistic regression, we'll multiply the $\theta$ vector with the X data, and as we have a non-zero bias  $\beta_0$ we'd like to have an X matrix whose first column is filled with ones.

\begin{align}
    X_{\small{with\ bias}} = \begin{pmatrix}
        1 & X_{1,0} & X_{2,0}\\
        1 & X_{1,1} & X_{2,1}\\
        &...&\\
        1 & X_{1,n} & X_{2,n}
        \end{pmatrix}
\end{align}

**Q3) Prepare the `X_with_bias` matrix (remember to use the `bin_X` data and not just `X`). Write a function called `predict` that takes in the parameter vector $\theta$ and the `X_with_bias` matrix and evaluates the logistic function for each of the samples.**

Hint 1: You recently learned how to initialize arrays in the `Numpy` notebook [at this link](https://nbviewer.org/github/tbeucler/2022_ML_Earth_Env_Sci/blob/main/Lab_Notebooks/S1_2_Numpy.ipynb). There are many ways to add a columns of 1 to `bin_X`, for instance using [`np.concatenate`](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html) or [`np.append`](https://numpy.org/doc/stable/reference/generated/numpy.append.html).

Hint 2:  To clarify, the function `predict` calculates $\hat{p}$ from $\beta$ and $\boldsymbol{X}$.

Hint 3: In practice, to calculate the logistic function for each sample, you may follow the equations [higher up in the notebook](#logit) and (1) calculate $t$ from $\beta$ and $\boldsymbol{X_{\mathrm{with\ bias}}}$ before (2) applying the logistic function $\sigma$ to $t$.

```python
# Prepare the X_with_bias matrix
n = bin_X.shape[0]
m = theta.shape[0] # theta is an arr of dim (3,1)
X_with_bias = np.ones((n, m)) # x_new is an array of dim (100,3) with ones
X_with_bias[:, [1,2]] = bin_X[:,:] # the first column is ones, the rest is == x
type(X_with_bias)
#print(X_with_bias[:10,:])
#print(X_with_bias.shape[1])
#print(theta.shape[0])
```

    numpy.ndarray

```python
# Write your function predict here
def predict(x,theta):
  # dimension for x (n,3)
  # dimension for theta (3,1)
  assert x.shape[1] == theta.shape[0], "matrix dim must agree"
  # Function that computes the logistic function.
  t = np.matmul(x,theta) # we multiply and obtain a (100,1) array
  #sigma=1/(1+np.exp(-t))
  sigma = logistic(t)
  return sigma #dim (n,1)
```

```python
p = predict(X_with_bias,theta)
log_loss(p, bin_y)
```

    329.36299914291806

**Q4) Now that you have a `predict` function, write a `gradient_calc` function that calculates the gradient for the logistic function.**

Hint 1: You'll have to feed `theta`, `X`, and `y` to the `gradient_calc` function.

Hint 2: You can use [this equation](#grad_eq) to calculate the gradient of the cost function.

```python
# Write your code here
def gradient_calc(x, theta, y):
  # dim for x (n,3)
  # dim for theta (3,1)
  # dim for y (n,1)
  Jderiv = np.zeros((3,1))
  assert y.shape[0] == x.shape[0]
  sigma = predict(x,theta)
  assert y.shape[0] == sigma.shape[0]
  aux1 = (sigma - y) * x[:,0]
  aux2 = (sigma - y) * x[:,1]
  aux3 = (sigma - y) * x[:,2]
  Jderiv = np.array([[np.mean(aux1)], [np.mean(aux2)], [np.mean(aux3)]])
  return Jderiv # the exit is a column vector, as theta is, of dim (3,1)
```

```python
print(X_with_bias.shape[0])
print(bin_y.shape[0])

#jderiv = np.zeros((3,1))
jderiv = gradient_calc(X_with_bias, theta,bin_y)

print(jderiv)
type(jderiv)

```

    100
    100
    [[0.49001071]
     [2.08099253]
     [0.64625795]]

    numpy.ndarray

We can now write a function that will train a logistic regression algorithm!

Your `logistic_regression` function needs to:
* Take in a set of training input/output data, validation input/output data, a number of iterations to train for, a set of initial parameters $\theta$, and a learning rate $\eta$
* At each iteration:
 * Generate a set of predictions on the training data. Hint: You may use your function `predict` on inputs `X_train` from the training set.
 * Calculate and store the loss function for the training data at each iteration. Hint: You may use your function `log_loss` on inputs `X_train` and outputs `y_train` from the training set.
 * Calculate the gradient. Hint: You may use your function `grad_calc`.
 * Update the $\theta$ parameters. Hint: You need to implement [this equation](#grad_descent).
 * Generate a set of predictions on the validation data using the updated parameters. Hint: You may use your function `predict` on inputs `X_valid` from the validation set.
 * Calculate and store the loss function for the validation data. Hint: You may use your function `log_loss` on inputs `X_valid` and outputs `y_valid` from the validation set.
 * Bonus: Calculate and store the accuracy of the model on the training and validation data as a metric!
* Return the final set of parameters $\theta$ & the stored training/validation loss function values (and the accuracy, if you did the bonus)

**Q5) Write the `logistic_regression` function**

```python
# Write your code here
def logistic_regression(X_train, y_train, X_valid, y_valid, iter, theta_ini, eta):
  theta = theta_ini
  J_test = np.zeros((iter,1))
  J_valid = np.zeros((iter,1))

  for k in range(iter):
    y_pred = predict(X_train, theta)
    J_test[k] = log_loss(y_pred, y_train)
    J_grad = gradient_calc(X_train, theta, y_train)
    theta = theta - eta * J_grad
    y_pred_valid = predict(X_valid, theta)
    J_valid[k] = log_loss(y_pred_valid, y_valid)

  Losses = np.array([J_test[:], J_valid[:]])
  beta = (theta[0], theta[1], theta[2])
  return  Losses, beta
```

**¡¡¡Important Note!!!**

The notebook assumes that you will return
1. a Losses list, where Losses[0] is the training loss and Losses[1] is the validation loss
2. a tuple with the 3 final coefficients ($\beta_0$, $\beta_1$, $\beta_2$)

The code for visualizing the bonus accuracy is not included - but it should be simple enough to do in a way similar to that which is done with the losses.

---------------------

Now that we have our logistic regression function, we're all set to train our algorithm! Or are we?

There's an important data step that we've neglected up to this point - we need to split the data into the train, validation, and test datasets.

```python
test_ratio = 0.2
validation_ratio = 0.2
total_size = len(X_with_bias)

test_size = int(total_size * test_ratio)
validation_size = int(total_size * validation_ratio)
train_size = total_size - test_size - validation_size

rnd_indices = rnd_gen.permutation(total_size)

X_train = X_with_bias[rnd_indices[:train_size]]
y_train = bin_y[rnd_indices[:train_size]]
X_valid = X_with_bias[rnd_indices[train_size:-test_size]]
y_valid = bin_y[rnd_indices[train_size:-test_size]]
X_test = X_with_bias[rnd_indices[-test_size:]]
y_test = bin_y[rnd_indices[-test_size:]]
```

Now we're ready!

**Q6) Train your logistic regression algorithm. Use 5000 iterations, $\eta$=0.1**

Hint: It's time to use the `logistic_regression` function you defined in Q5.

```python
# Complete the code
#type(theta)
losses, coeffs = logistic_regression(X_train, y_train, X_valid, y_valid, 5000, theta, 0.1)
print(losses)
print(coeffs)
```

    [[[209.71490642]
      [169.38428439]
      [130.14530739]
      ...
      [247.58181913]
      [247.62933134]
      [247.67684265]]

     [[ 39.36061763]
      [ 30.48010306]
      [ 22.39241391]
      ...
      [ 75.79586996]
      [ 75.81055496]
      [ 75.82524016]]]
    (array([-37.58908749]), array([1.3252815]), array([20.75106617]))

```python
type(losses)
coeffs = np.asarray(coeffs)
print(coeffs)
type(coeffs)
```

    [[-37.58908749]
     [  1.3252815 ]
     [ 20.75106617]]

    numpy.ndarray

Let's see how our model did while learning!

```python
# Produce the Loss Function Visualization Graphs
fig, ax = plt.subplots(figsize=(18,8))
ax.plot(losses[0], color='blue', label='Training', linewidth=3);
ax.plot(losses[1], color='orange', label='Validation', linewidth=3);
ax.legend();
ax.set_ylabel('Log Loss')
ax.set_xlabel('Iterations')
ax.set_title('Loss Function Graph')
ax.autoscale(axis='x', tight=True)
fig.tight_layout();

# Let's get predictions from our model for the training, validation, and testing
# datasets
y_hat_train = (predict(X_train, coeffs)>=.5).astype(int)
y_hat_valid = (predict(X_valid, coeffs)>=.5).astype(int)
y_hat_test = (predict(X_test, coeffs)>=.5).astype(int)

y_sets = [ [y_hat_train, y_train],
           [y_hat_valid, y_valid],
           [y_hat_test, y_test] ]

def accuracy_score(y_hat, y):
    assert(y_hat.size==y.size)
    return (y_hat == y).sum()/y.size

[accuracies.append(accuracy_score(y_set[0],y_set[1])) for y_set in y_sets]

printout= (f'Training Accuracy:{accuracies[0]:.1%} \n'
           f'Validation Accuracy:{accuracies[1]:.1%} \n')

# Add the testing accuracy only once you're sure that your model works!

print(printout)
```

Congratulations on training a logistic regression algorithm from scratch! Once you're done with the upcoming environmental science applications notebook, feel free to come back to take a look at the challenges 😀

## Challenges

* **C1)** Add L2 Regularization to training function

* **C2)** Add early stopping to the training algorithm! Stop training when the accuracy is >=90%

* **C3)** Implement a softmax regression model (It's multiclass logistic regression 🙂)