###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2017 L.A. Barba, N.C. Clementi

# Bird's-eye view of mechanical vibrations

Welcome to **Lesson 4** of the third module in _Engineering Computations_. This course module is dedicated to studying the dynamics of change with computational thinking, Python and Jupyter.
The first three lessons give you a solid footing to tackle problems involving motion, velocity, and acceleration. They are:

1. [Lesson 1](http://go.gwu.edu/engcomp3lesson1): Catch things in motion
2. [Lesson 2](http://go.gwu.edu/engcomp3lesson2): Step to the future
3. [Lesson 3](http://go.gwu.edu/engcomp3lesson3): Get with the oscillations

You learned to compute velocity and acceleration from position data, using numerical derivatives, and to capture position data of moving objects from images and video. For physical contexts we used free-fall of a ball, and projectile motion. Then you faced the opposite challenge: computing velocity and position from acceleration data, leading to the idea of stepping forward in time to solve a differential equation.

Our general approach combines these key ideas: (1) turning a second-order differential equation into a system of first-order equations; (2) writing the system in vector form, and the solution in terms of a state vector; (3) designing a code to obtain the solution using separate functions to compute the derivatives of the state vector, and to step the system in time with a chosen scheme (e.g., Euler, Euler-Cromer, Runge-Kutta). It's a rock-steady approach that will serve you well!

In this lesson, you'll get a broader view of applying your new-found skills to learn about mechanical vibrations: a classic engineering problem. You'll study general spring-mass systems with damping and a driving force, and appreciate the diversity of behaviors that arise. We'll end the lesson presenting a powerful method to study dynamical systems: visualizing direction fields and trajectories in the phase plane.

Are you ready? Start by loading the Python libraries that we know and love.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
pyplot.rc('font', family='serif', size='14')
```

## General spring-mass system

The simplest mechanical oscillating system is a mass $m$ attached to a spring, without friction. We discussed this system in the [previous lesson](http://go.gwu.edu/engcomp3lesson3). In general, though, these systems are subject to friction—represented by a mechanical damper—and a driving force. Also, the spring's restoring force could be a nonlinear function of position, $k(x)$.

#### General spring-mass system, with driving and damping.

Newton's law applied to the general (driven, damped, nonlinear) spring-mass system is:

\begin{equation}
 m \ddot{x} = F(t) -b(\dot{x}) - k(x)
\end{equation}

where
* $F(t)$ is the driving force
* $b(\dot{x})$ is the damping force
* $k(x)$ is the restoring force, possibly nonlinear

Written as a system of two differential equations, we have:

\begin{eqnarray}
\dot{x} &=& v, \nonumber\\
\dot{v} &=& \frac{1}{m} \left( F(t) - k(x) - b(v) \right).
\end{eqnarray}

With the state vector,
\begin{equation}
\mathbf{x} = \begin{bmatrix}
x \\ v
\end{bmatrix},
\end{equation}

the differential equation in vector form is:

\begin{equation}
\dot{\mathbf{x}} = \begin{bmatrix}
v \\ \frac{1}{m} \left( F(t) - k(x) - b(v) \right)
\end{bmatrix}.
\end{equation}

In this more general system, the time variable could appear explicitly on the right-hand side, via the driving function $F(t)$. We'll need to adapt the code for the time-stepping function to take the time as an additional argument.

The `euler_cromer()` function we defined in the previous lesson took three arguments: `state, rhs, dt`—the state vector, the Python function computing the right-hand side of the differential equation, and the time step. Let's re-work that function now to take an additional `time` variable, which also gets used in the `rhs` function.

```python
# new version of the function, taking time as explicit argument
def euler_cromer(state, rhs, time, dt):
    '''Update a state to the next time increment using Euler-Cromer's method.

    Arguments
    ---------
    state : state vector of dependent variables
    rhs   : function that computes the RHS of the DE, taking (state, time)
    time  : float, time instant
    dt    : float, time step

    Returns
    -------
    next_state : state vector updated after one time increment'''

    mid_state = state + rhs(state, time)*dt # Euler step
    mid_derivs = rhs(mid_state, time)       # update derivatives

    next_state = numpy.array([mid_state[0], state[1] + mid_derivs[1]*dt])

    return next_state
```

### Case with linear damping

Let's look at the behavior of a system with linear restoring force, linear damping, but no driving force: $k(x)= kx$, $b(v)=bv$, $F(t)=0$.
The differential system is now:

\begin{equation}
\dot{\mathbf{x}} = \begin{bmatrix}
v \\ \frac{1}{m} \left( - kx - bv \right)
\end{bmatrix}.
\end{equation}

Now we need to write a function to compute the right-hand side (derivatives) for this system.
Even though the system does not explicitly use the time variable in the right-hand side, we still include `time` as an argument to the function, so that it is consistent with our new design for the `euler_cromer()` step. We include `time` in the arguments list, but it is not used inside the function code. It's thus a good idea to specify a _default value_ for this argument by writing `time=0` in the arguments list: that will allow us to also call the function leaving the `time` argument blank, if we wanted to (in which case, it will automatically be assigned its default value of 0).
Another option for the default value is `time=None`. It doesn't matter because the variable is not used inside the function!

```python
def dampedspring(state, time=0):
    '''Computes the right-hand side of the spring-mass differential
    equation, with linear damping.

    Arguments
    ---------
    state : state vector of two dependent variables
    time : float, time instant

    Returns
    -------
    derivs: derivatives of the state vector
    '''

    derivs = numpy.array([state[1], 1/m*(-k*state[0]-b*state[1])])
    return derivs
```

Let's try it!
The following example is from section 4.3.9 of Ref. [1] (an open-access text!).
We set the model parameters, the initial conditions, and the time-stepping conditions.
Then we initialize the numerical solution array `num_sol`, and call the `euler_cromer()` function in the `for` statement.
Notice that we pass the time instant `t[i]` to the function's `time` argument (which will allow us to use the same calling signature when we solve for a system with driving force).

```python
m = 1.0
k = 1.0
b = 0.3
```

```python
x0 = 1    # initial position
v0 = 0    # initial velocity
```

```python
T = 12*numpy.pi
N = 5000
dt = T/N

t = numpy.linspace(0, T, N)
```

```python
num_sol = numpy.zeros([N,2]) #initialize solution array

#Set intial conditions
num_sol[0,0] = x0
num_sol[0,1] = v0
```

```python
for i in range(N-1):
    num_sol[i+1] = euler_cromer(num_sol[i], dampedspring, t[i], dt)
```

Time to plot the solution—in our plot of position versus time below, notice that we added a line with [`pyplot.figtext()`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.figtext.html?highlight=matplotlib%20pyplot%20figtext#matplotlib.pyplot.figtext) at the end. This command adds a custom text to the figure: we use it to print the values of the spring-mass model parameters corresponding to the plot. See how we print the parameter values in the text string? We used Python's string formatter, which you learned about in [Module 2 Lesson 1](http://go.gwu.edu/engcomp2lesson1).
If we were to re-run the solution with different model parameters, re-executing the code in this cell would update the plot and the text with the proper values. (We don't want to rely on manually changing the text, as that is error prone!)

```python
fig = pyplot.figure(figsize=(6,4))

pyplot.plot(t, num_sol[:, 0], linewidth=2, linestyle='-')
pyplot.xlabel('Time [s]')
pyplot.ylabel('Position, $x$ [m]')
pyplot.title('Damped spring-mass system with Euler-Cromer method.\n')
pyplot.figtext(0.1,-0.1,'$m={:.1f}$, $k={:.1f}$, $b={:.1f}$'.format(m,k,b));
```

The result above shows that the oscillations die down over a few periods: the oscillations are _damped_ over time.
And our plot looks pretty close to [Fig. 4.27](https://link.springer.com/chapter/10.1007%2F978-3-319-32428-9_4#Fig27) of Ref. [1], as it should.

### Case with sinusoidal driving, and damping

Suppose now that an external force of the form $F(t) = A \sin(\omega t)$ drives the system. This is a typical situation in mechanical systems. Let's find out what a system like that behaves like. The example below comes from section 4.3.10 of Ref. [1].

We're showy, so we decided to use the Unicode character for the Greek letter $\omega$ in the code… because we can!
With a handy table of [Unicode for greek letters](https://gist.github.com/beniwohli/765262), you can pick a symbol code, type it into a code cell, and out comes the symbol. Then, it's a copy-and-paste job to reuse the symbol in the code. And using greek letters for some variable names is very chic.

```python
u'\u03C9'
```

    'ω'

```python
A = 0.5  # parameter values from example in 4.3.10 of Ref. [1]
ω = 3
```

More than showy, we're snazzy, and so we build a one-line function using the [`lambda`](https://docs.python.org/3/reference/expressions.html#lambda) keyword.
It's just too cool.
In Python, you can create a small function in one line using the assignment operator `=`, followed by the `lambda` keyword, then a statement of the form `arguments: expression`—in our case, we have the single argument `time`, and the expression is the sinusoidal driving.
The sine mathematical function is avaible to us from the [`math` library](https://docs.python.org/3/library/math.html). Check it out.

```python
from math import sin
F = lambda time: A*sin(ω*time)
```

This is really a function: we can call `F()` at any point in our code, passing a value of time, and it will output the result of $F(t) = A \sin(\omega t)$.

Now, let's write the right-hand side function of derivatives for the driven spring-mass system (with damping). Notice that we use the lambda function `F()` inside this new function, and the `time` variable explicitly as the argument to `F()`. Some powerful Python kung fu!

```python
def drivenspring(state, time):
    '''Computes the right-hand side of the spring-mass differential
    equation, with sinusoidal driving and linear damping.

    Arguments
    ---------
    state : state vector of two dependent variables
    time : float, time instant

    Returns
    -------
    derivs: derivatives of the state vector
    '''

    derivs = numpy.array([state[1], 1/m*(F(time)-k*state[0]-b*state[1])])
    return derivs
```

Here is where the power of our code design becomes clear: solving the differential equation via time-stepping inside a `for` statement looks just like before, with the only difference being that we pass another right-hand-side function of derivatives.
The code cell below solves the driven spring-mass system with the same model parameters we used for the damped system without driving.

```python
for i in range(N-1):
    num_sol[i+1] = euler_cromer(num_sol[i], drivenspring, t[i], dt)
```

```python
fig = pyplot.figure(figsize=(6,4))

pyplot.plot(t, num_sol[:, 0], linewidth=2, linestyle='-')
pyplot.xlabel('Time [s]')
pyplot.ylabel('Position, $x$ [m]')
pyplot.title('Driven spring-mass system with Euler-Cromer method.\n')
pyplot.figtext(0.1,-0.1,'$m={:.1f}$, $k={:.1f}$, $b={:.1f}$, $A={:.1f}$, $\omega={:.1f}$'.format(m,k,b,A,ω));
```

And our result looks just like [Fig. 4.28](https://link.springer.com/chapter/10.1007%2F978-3-319-32428-9_4#Fig28) of Ref. [1], as it should. You can see that the system starts out dominated by the spring-mass oscillations, which get damped over time and the effect of the external driving becomes visible, and the sinusoidal driving is all that is left in the end.

##### Exercise:

* Experiment with different values of the driving-force amplitude, $A$, and frequency, $\omega$.
* Swap the sine driving for a cosine, and see what happens.

An interesting behavior occurs when the damping is low enough and the frequency of the driving force coincides with the natural frequency of the mass-spring system, $\sqrt{k/m}$: **resonance**.

Try these parameters:

```python
ω = 1
b = 0.1
```

```python
for i in range(N-1):
    num_sol[i+1] = euler_cromer(num_sol[i], drivenspring, t[i], dt)
```

```python
fig = pyplot.figure(figsize=(6,4))

pyplot.plot(t, num_sol[:, 0], linewidth=2, linestyle='-')
pyplot.xlabel('Time [s]')
pyplot.ylabel('Position, $x$ [m]')
pyplot.title('Driven spring-mass system with Euler-Cromer method.\n')
pyplot.figtext(0.1,-0.1,'$m={:.1f}$, $k={:.1f}$, $b={:.1f}$, $A={:.1f}$, $\omega={:.1f}$'.format(m,k,b,A,ω));
```

As you can see, the amplitude of the oscillations grow over time! (Compare the vertical axis of this plot with the previous one.) Our result matches with [Fig. 4.29](https://link.springer.com/chapter/10.1007%2F978-3-319-32428-9_4#Fig29) of Ref. [1].

## Solutions on the phase plane

The spring-mass system, as you see, can behave in various ways. If the spring is linear, and there is no damping or driving (like in the previous lesson), the motion is periodic. If we add damping, the oscillatory motion decays over time. With driving, the motion can be rather more complicated, and sometimes can exhibit resonance.

Each of these types of motion is represented by corresponding solutions to the differential system, dictated by the model parameters and the initial conditions.

How could we get a sense for all the types of solutions to a differential system?
A powerful method to do this is to use the _phase plane_.

A system of two first-order differential equations:

\begin{eqnarray}
\dot{x}(t) &=& f(x, y) \\
\dot{y}(t) &=& g(x, y)
\end{eqnarray}

with state vector

\begin{equation}
\mathbf{x} = \begin{bmatrix}
x \\ y
\end{bmatrix},
\end{equation}

is called a _planar autonomous system_: planar, because the state vector has two components; and autonomous (self-generating), because the time variable does not explicitly appear on the right-hand side
(which wouldn't apply to the driven spring-mass system).

For initial conditions $\mathbf{x}_0=(x_0, y_0)$, the system has a unique solution $\mathbf{x}(t)=\left(x(t), y(t)\right)$. This solution can be represented by a planar curve on the $xy$-plane—the **phase plane**—and is called a _trajectory_ of the system.

On the phase plane, we can plot a **direction (slope) field** by generating a uniform grid of points $(x_i, y_j)$ in some chosen range $(x_\text{min}, x_\text{max})\times(y_\text{min}, y_\text{max})$, and drawing small line segments representing the direction of the vector field $(f(x,y), g(x,y)$ on each point.

Let's draw a direction field for the damped spring-mass system, and include a solution trajectory. We copied the whole problem set-up below, to get a solution all in one code cell, for easy trial with different parameter choices.

```python
m = 1
k = 1
b = 0.3

x0 = 3    # initial position
v0 = 3    # initial velocity

T = 12*numpy.pi
N = 5000
dt = T/N

t = numpy.linspace(0, T, N)
num_sol = numpy.zeros([N,2]) #initialize solution array

#Set intial conditions
num_sol[0,0] = x0
num_sol[0,1] = v0

for i in range(N-1):
    num_sol[i+1] = euler_cromer(num_sol[i], dampedspring, t[i], dt)
```

To choose a range for the plotting area of the direction field, let's look at the maximum values of the solution array.

```python
numpy.max(num_sol[:,0])
```

    4.0948277569088525

```python
numpy.max(num_sol[:,1])
```

    3.0

With that information, we choose the plotting area as $(-4,4)\times(-4,4)$. Below, we'll create an array named `coords` to hold the positions of mesh lines on each coordinate direction. Here, we pick 11 mesh points in each direction.

Then, we'll call the very handy [`meshgrid()`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.meshgrid.html) function of NumPy—you should definitely study the documentation and use pen and paper to diligently figure out what it does!

The outputs of `meshgrid()` are two matrices holding the $x$ and $y$ coordinates, respectively, of points on the grid. Combined, these two matrices give the coordinate pairs of every grid point where we'll compute the direction field.

```python
coords = numpy.linspace(-4,4,11)
X, Y = numpy.meshgrid(coords, coords)
```

Look at the vector form of the differential system again… with our two matrices of coordinate values for the grid points, we could compute the vector field on all these points in one go using array operations:

```python
F = Y
G = 1/m * (-k*X -b*Y)
```

Matplotlib has a type of plot called [`quiver`](https://matplotlib.org/examples/pylab_examples/quiver_demo.html) that draws a vector field on a plane. Let's try it out using the vector field we computed above.

```python
fig = pyplot.figure(figsize=(7,7))

pyplot.quiver(X,Y, F,G);
```

OK, that's not bad. The arrows on each grid point represent vectors $(f(x,y), g(x,y))$, computed from the right-hand side of the differential equation.

_What are the axes on this plot?_ Well, they are the components of the state vector—which for the spring-mass system are _position_ and _velocity_. The vector field looks like a "flow" going around the origin, the values of position and velocity oscillating around. If you imagine an initial condition represented by a coordinate pair $(x_0,y_0)$, the solution trajectory would follow along the arrows, spiraling around the origin, while slowly approaching it.

We'd like to visualize a trajectory on the vector-field plot, and also improve it in a few ways. But before that, Python will astonish you with a splendid fact: you can also compute the vector field on the grid points by calling the function `dampedspring()`, passing as argument a list made of the matrices `X` and `Y`.

_Why does this work?_ Study the function and think!

```python
F, G = dampedspring([X,Y])
```

The default behavior of `quiver` is to scale the vectors (arrows) with the magnitude, but direction fields are usually drawn using line segments of equal length. Also by default, the vectors are drawn _starting at_ the grid points, while direction fields ususally _center_ the line segments. We can improve our plot using by _scaling_ the vectors by their magnitude, and using the `pivot='mid'` option on the plot. A little transparency is also nice.

To plot the improved direction field below, we drew ideas from a tutorial available online, see Ref. [2]. To compute the magnitude of the vectors, we use the [`numpy.hypot()`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.hypot.html) function, which returns the triangle hypotenuse of two right-angled sides.

We should also add axis labels and a title!

```python
M = numpy.hypot(F,G)
M[ M == 0] = 1      # to avoid zero-division
F = F/M
G = G/M

fig = pyplot.figure(figsize=(7,7))

pyplot.quiver(X,Y, F,G, pivot='mid', alpha=0.5)
pyplot.plot(num_sol[:,0], num_sol[:,1], color= '#0096d6', linewidth=2)
pyplot.xlabel('Position, $x$, [m]')
pyplot.ylabel('Velocity, $v$, [m/s]')
pyplot.title('Direction field for the damped spring-mass system\n')
pyplot.figtext(0.1,0,'$m={:.1f}$, $k={:.1f}$, $b={:.1f}$'.format(m,k,b));
```

And just for kicks, let's re-do everything with zero damping:

```python
m = 1
k = 1
b = 0

x0 = 3    # initial position
v0 = 3    # initial velocity

T = 12*numpy.pi
N = 5000
dt = T/N

t = numpy.linspace(0, T, N)
num_sol = numpy.zeros([N,2]) #initialize solution array

#Set intial conditions
num_sol[0,0] = x0
num_sol[0,1] = v0

for i in range(N-1):
    num_sol[i+1] = euler_cromer(num_sol[i], dampedspring, t[i], dt)

F, G = dampedspring([X,Y])

M = numpy.hypot(F,G)
M[ M == 0] = 1      # to avoid zero-division
F = F/M
G = G/M

fig = pyplot.figure(figsize=(7,7))

pyplot.quiver(X,Y, F,G, pivot='mid', alpha=0.5)
pyplot.plot(num_sol[:,0], num_sol[:,1], color= '#0096d6', linewidth=2)
pyplot.xlabel('Position, $x$, [m]')
pyplot.ylabel('Velocity, $v$, [m/s]')
pyplot.title('Direction field for the un-damped spring-mass system\n')
pyplot.figtext(0.1,0,'$m={:.1f}$, $k={:.1f}$, $b={:.1f}$'.format(m,k,b));

```

##### Challenge task

* Write a function to draw direction fields as above, taking as arguments the right-hand-side derivatives function, and lists containing the plot limits and the number of grid lines in each coordinate direction.

* Write some code to capture mouse clicks on the direction field, following what you learned in [Lesson 1](http://go.gwu.edu/engcomp3lesson1) of this module.

* Use the captured mouse clicks as initial conditions and obtain the corresponding trajectories by solving the differential system, then make a new plot showing the trajectories with different colors.

## What we've learned

* General spring-mass systems have several behaviors: periodic in the undamped case, decaying oscillations when damped, complex oscillations when driven.
* Resonance appears when the driving frequency matches the natural frequency of the system.
* We can add formatted strings in figure titles, labels and added text.
* The `lambda` keyword builds one-line Python functions.
* The `meshgrid()` function of NumPy is handy for building a grid of points on a plane.
* State vectors of a differential system live on the _phase plane_.
* Solutions of the differential system (given initial conditions) are _trajectories_ on the phase plane.
* Trajectories for the undamped spring-mass system are circles; in the damped case, they are spirals toward the origin.

## References

1. Linge S., Langtangen H.P. (2016) Solving Ordinary Differential Equations. In: Programming for Computations - Python. Texts in Computational Science and Engineering, vol 15. Springer, Cham, https://doi.org/10.1007/978-3-319-32428-9_4, open access and reusable under [CC-BY-NC](http://creativecommons.org/licenses/by-nc/4.0/) license.
V
2. [Plotting direction fields and trajectories in the phase plane](http://scipy-cookbook.readthedocs.io/items/LoktaVolterraTutorial.html?highlight=direction%20fields#Plotting-direction-fields-and-trajectories-in-the-phase-plane), as part of the Lotka-Volterra tutorial by Pauli Virtanen and Bhupendra, in the _SciPy Cookbook_.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


### Instructions

Please be patient while waiting for the environment to load; it may take a few minutes.
Once the notebook has finished loading, in the top bar, select `Kernel`--> `Restart Kernel and Run All Cells`.
Once the notebook has finished running, you should see a plot with sliders appear at the bottom.

If you make any changes in a coding cell, rerun the notebook by `Run` > `Run Selected Cell and All Below`

Enjoy!

```python
# Importing libraries and dependencies
import sys
sys.path.append('python/')
import NGC5533_functions as funcNGC5533        # Functions for the components of NGC 5533
import NGC5533_traced_curves as dataNGC5533    # Traced data of Galaxy NGC 5533
import NGC5533_fitting as fitNGC5533           # Fitting parameters for best fit values
import dataPython as dp

import numpy as np
import matplotlib.pyplot as plt                # Plotting
%matplotlib inline

import scipy.integrate as si                   # Integration
from ipywidgets import interactive, fixed, FloatSlider, HBox, Layout, Button, Label, Output, VBox
from IPython.display import display, clear_output
from IPython.display import Javascript

import scipy.stats as stats
import warnings
warnings.filterwarnings("ignore")              # Ignore warnings
```

# Introduction to Dark Matter

If luminous matter was the only matter in spiral galaxies, then the rotation curves would resemble that of planet-like, or Keplerian, motion. Since the rotation curves of some spiral galaxies are flat, we can infer that there is "missing" matter that is not interacting with light but affecting the velocity of stars and gas. We call this missing matter "dark matter."

According to Newton's law of gravitation, gravitational force on an object is proportional to the mass interacting with it, and therefore the motion of matter in a galaxy is also proportional to this mass.
Setting the gravitational force and the centripetal force equal, the rotational velocity can be computed in terms of mass ($M$) and distance ($r$) from the center.

>__Newton's Law of Gravitation__:

    \begin{equation}
    F_g = G \frac{Mm}{r^2}
    \end{equation}

>__Centripetal Force__:

    \begin{equation}
    F_c = \frac{mv^2}{r}
    \end{equation}

>__Rotational velocity__:

    \begin{equation}
    v = \sqrt{\frac{GM}{r}}
    \end{equation}

Since visible matter can be measured through photometry, a theoretical rotation curve can be computed applying the above equations.

However, as you will discover in the following exercises, this theoretical curve does not always agree with the measured velocities.

## Interactive widget with variable Dark Matter component - Galaxy: NGC 5533

In the following activity, you can experiment adding or removing Dark Matter and investigating how the total velocity of stars and gases change in a galaxy.

### Measured data points

In the code below, you will import the measured data points for the NGC 5533 galaxy. These are measured velocities of stars and gas in the galaxy as a function of radius. The data points were traced from Figure 4. in Noordermeer (2008).

```python
# Import measured velocity data
data = dp.getXYdata_wXYerr('data/NGC5533/noord-120kpc-datapoints.txt')
r_dat = np.asarray(data['xx'])              # radius
v_dat = np.asarray(data['yy'])              # velocity
v_err0 = np.asarray(data['ex'])             # error in radius
v_err1 = np.asarray(data['ey'])             # error in velocity
```

```python
# Plot
plt.figure(figsize=(15,6))                                               # size of the galaxy
plt.errorbar(r_dat,v_dat,yerr=v_err1, marker='o', markersize=8, \
             ecolor='gray',color='gray', linestyle='none', linewidth=2)  # plot datapoints with errorbars
plt.xlabel('radius (kpc)',size=12)                                       # label x-axis
plt.ylabel('velocity (km/s)',size=12)                                    # label y-axis
plt.title(str('Observed Velocity of NGC 5533'), size=14)                 # title of the plot
plt.xlim(0,np.max(r_dat)+0.2)                                            # range of the x-axis
plt.ylim(0,np.max(v_dat)+100)                                            # range of the y-axis
plt.show()                                                               # show the plot
```

### Galactic components

The components of a galaxy are categorized based on how stars and gas are distributed. You will learn more about these components in the notebook `04_Plotting_Rotation_Curves.ipynb`. In the case of NGC 5533, we can define components consisting of a central supermassive black hole, a central bulge, a flat disk, atomic gas, and a dark matter halo. To calculate the total velocity or the rotation curve of a galaxy, these components are added in quadrature:

>__Total velocity__:

    \begin{equation}
    v_{total}(r) = \sqrt{\lvert v_{gas}\rvert v_{gas} + \Upsilon _{bulge} \lvert v_{bulge}\rvert v_{bulge} + \Upsilon _{disk} \lvert v_{disk}\rvert v_{disk} + \lvert v_{halo}\rvert v_{halo}}
    \end{equation}

Let's import the relevant functions or traced curves from the `NGC5533_functions.py` library:

```python
# Black hole component, traced curve
def blackhole(r,Mbh):
    return funcNGC5533.bh_v(r,Mbh,load=False)

# Bulge component, traced curve
def bulge(r,bpref):
    return bpref*funcNGC5533.b_v(r,load=True)

# Disk component, traced curve
def disk(r,dpref):
    return dpref*funcNGC5533.d_thief(r)

# Gas component, traced curve
def gas(r,gpref):
    return gpref*funcNGC5533.g_thief(r)

# Dark matter component, traced curve
def halo(r,rc,rho0):
    return funcNGC5533.h_v(r,rc,rho0,load=False)

# Total velocity containing all components, added in quadrature
def total_all(r,Mbh,bpref,dpref,gpref,rc,rho0):
    total = np.sqrt(blackhole(r,Mbh)**2               # black hole
                    + bulge(r,bpref)**2               # bulge
                    + disk(r,dpref)**2                # disk
                    + gas(r,gpref)**2                 # gas
                    + halo(r,rc,rho0)**2)             # dark matter halo
    return total

# Total velocity of baryonic or luminous matter (no dark matter component), added in quadrature
def total_bary(r,Mbh,bpref,dpref,gpref):
    total = np.sqrt(blackhole(r,Mbh)**2               # black hole
                    + bulge(r,bpref)**2               # bulge
                    + disk(r,dpref)**2                # disk
                    + gas(r,gpref)**2)                # gas
    return total
```

### Parameters

The scaling parameters for each component can be found by fitting the total velocity to the measured data points. Lets import the calculated fitting parameters for our widget from the `NGC5533_fitting.py` library.

```python
# Scaling parameters
best_Mbh = fitNGC5533.f_Mbh
best_bpref = fitNGC5533.f_bpref
best_dpref = fitNGC5533.f_dpref
best_gpref = fitNGC5533.f_gpref
best_rc = fitNGC5533.f_rc
best_rho0 = fitNGC5533.f_rho00
```

### Total Dark Matter mass

Assuming the density of the Dark Matter halo is isothermal, meaning the temperature remains constant, we can calculate the total expected Dark Matter mass enclosed in a given radius. Starting with the equation for isothermal density as a function of radius, we can calculate the mass:

>__Isothermal Density__:

    \begin{equation}
    \rho(r) = \rho_{0} \big[1 + \frac{r}{r_c} \big ]^{-1}
    \end{equation}

where:

        $G$ = gravitational constant

        $\rho_0$ = central mass density (in solar mass/$\rm kpc^3$)

        $r_c$ = core radius (in kpc)

__Mass as a function of radius__:

    \begin{equation}
    M(r) = 4 \pi r^2 \rho(r)
    \end{equation}

Let's define some functions for the above equations, and then plot the distribution of mass of NGC 5533:

```python
# Equation for isothermal density
def density_iso(r,rc,rho0):
    density = rho0 * (1 + (r/rc)**2)**(-1)
    return density
```

```python
# Equation for mass as a function of radius
def mass_function(r,rc,rho0):
    mass = 4 * np.pi * r**2 * density_iso(r,rc,rho0)
    return mass
```

```python
# Plot to visualize the mass distribution in the NGC 5533 galaxy for radii between 0 and 20 kpc
r = np.linspace(np.min(r_dat),np.max(r_dat),1000)
plt.figure(figsize=(10,6))
plt.plot(r,mass_function(r,best_rc,best_rho0))
plt.xlabel('radius (kpc)',size=12)
plt.ylabel('mass ($M_{sun}$)',size=12)
plt.title(str('Mass of Dark Matter as a function of radius - NGC 5533'), size=14)
plt.xlim(0,20)
plt.ylim(0,1e10)
plt.show()
```

Integrate to calculate the total mass enclosed.

>__Total mass enclosed__:

    \begin{equation}
    M_{total} = \int_{0}^{r} M(r) dr
    \end{equation}

```python
# Integrate to calculate total mass enclosed
TotalMass = lambda rc,rho0: si.quad(mass_function, 0, np.max(r_dat), args=(rc,rho0))[0]

# Print total mass of Dark Matter in NGC 5533
print("The total mass of Dark Matter in the galaxy NGC 5533 is about {:.3e} solar masses.".format(TotalMass(best_rc,best_rho0)))
```

    The total mass of Dark Matter in the galaxy NGC 5533 is about 8.867e+11 solar masses.

### Define plotting function and sliders for interactive widget

The interactive widget has two adjustable sliders where you can change the radius and the mass density of the halo. All other parameters are fixed to focus on the dark matter componet only. The next two cells of code define functions related to the widget, but do not generate the widget itself.

```python
# Plotting function
def f(rc,rho0):

    # Define radius
    r = np.linspace(np.min(r_dat),np.max(r_dat),1000)

    # Plot
    plt.figure(figsize=(11,7))
    plt.xlim(0,np.max(r_dat)+0.2)
    plt.ylim(0,np.max(v_dat)+100)

    plt.errorbar(r_dat,v_dat,yerr=v_err1,fmt='bo',label='Data')     # Measured data points
    plt.plot(r,blackhole(r,best_Mbh),label=("Central Supermassive Black Hole"),color='black') # Black hole component
    plt.plot(r,bulge(r,best_bpref),label=("Bulge"),color='orange')       # Bulge component
    plt.plot(r,disk(r,best_dpref),label=("Disk"),color='purple')         # Disk component
    plt.plot(r,gas(r,best_gpref),label=("Gas"),color='blue')             # Gas component
    plt.plot(r,halo(r,rc,rho0),label=("Halo"),color='green')        # Dark matter halo component
    plt.plot(r,total_all(r,best_Mbh,best_bpref,best_dpref,best_gpref,rc,rho0),label=("Total Curve"),color='red')    # Total velocity with dark matter
    plt.plot(r,total_bary(r,best_Mbh,best_bpref,best_dpref,best_gpref),label=("Luminous Matter"),linestyle='--')    # Total velocity without dark matter
    plt.fill_between(r,dataNGC5533.greyb_bottom(r),dataNGC5533.greyb_top(r),color='#dddddd',label="Confidence Band")    # Confidence band
    plt.title("Interactive Rotation Curve - Galaxy: NGC 5533",fontsize=20)
    plt.xlabel("Radius (kpc)")
    plt.ylabel("Velocity (km/s)")
    plt.legend(bbox_to_anchor=(1,1), loc="upper left")              # Put legend outside of the plot

    # Chi squared and reduced chi squared
    # Residuals
    residuals = v_dat - total_all(r_dat,best_Mbh,best_bpref,best_dpref,best_gpref,rc,rho0)
    # Error
    error = np.sqrt(v_err1**2 + dataNGC5533.band**2)
    # Chi squared
    chisquared = np.sum(residuals**2/error**2)
    # Degrees of freedom
    dof = len(r_dat) - 6                 # number of degrees of freedom = number of observed data - number of fitting parameters
    # Reduced chi squared
    reducedchisquared = chisquared / dof

    # Annotation
    props = dict(boxstyle='round', facecolor='white', alpha=0.5)
    plt.text(82,373,r"$\chi^2$: {:.5f}".format(chisquared)+'\n'+r"Reduced $\chi^2$: {:.5f}".format(reducedchisquared),bbox=props,size=10)
    props = dict(boxstyle='round', facecolor='white', alpha=0.5)
    plt.text(69.5,350,r"Total Dark Matter Mass: {:.2e} $M_\odot$".format(TotalMass(rc,rho0)),bbox=props,size=10)
    plt.annotate('Data source: E. Noordermeer. The rotation curves of flattened Sérsic bulges. MNRAS,385(3):1359–1364, Apr 2008',
            xy=(0, 0), xytext=(0,5),
            xycoords=('axes fraction', 'figure fraction'),
            textcoords='offset points',
            size=10, ha='left', va='bottom')

    plt.show()
```

```python
# Appearance of widget
style = {'description_width': 'initial'}
layout = {'width':'600px'}

# Sliders for the widget
rc = FloatSlider(min=0.1, max=5, step=0.1,
                    value=best_rc,
                    description='Halo Core Radius [kpc]',
                    readout_format='.2f',
                    orientation='horizontal',
                    style=style, layout=layout)
rho0 = FloatSlider(min=0, max=1e9, step=1e7,
                    value=best_rho0,
                    description=r'Halo Central Mass Density [$M_{\odot} / kpc^3$]',
                    readout_format='.2e',
                    orientation='horizontal',
                    style=style, layout=layout)

# Interactive widget
def interactive_plot(f):
    interact = interactive(f,
                           rc = rc,
                           rho0 = rho0,
                           continuous_update=False)
    return interact

# Button to revert back to Best Fit
button = Button(
    description="Best Fit",
    button_style='warning',
    icon='check')
out = Output()

def on_button_clicked(_):
    rc.value = best_rc
    rho0.value = best_rho0
button.on_click(on_button_clicked)
```

### What do rotation curves look like with and without Dark Matter?

In this activity, you can visualize how important Dark Matter is to accurately describe observed data (marked in blue points with error bars below).

How much Dark Matter is in a given galaxy to account for our observations?

Activity 1)

>Use the sliders below to modify the halo parameters and investigate how incorporating a Dark Matter component allows us to more accurately match our observations. In order to determine if a model is a good fit to observed data, we use a statistic called reduced $\chi^2$. A reduced $\chi^2$ close to 1 is generally considered a good fit. A reduced $\chi^2$ far higher than 1 is not a good fit. And a reduced $\chi^2$ which is a lot lower than 1 suggests a poor theoretical model (that is, the model probably doesn't accurately describe the real universe or it is simply connecting the measured data points). As you adjust the sliders, what happens to the total rotation curve (in red) as reduced $\chi^2$ approaches 1?

```python
#NBVAL_IGNORE_OUTPUT
#Because the figure doesn't save to the repository correctly.
# Widget
VBox([button,out,interactive_plot(f)])
```

    VBox(children=(Button(button_style='warning', description='Best Fit', icon='check', style=ButtonStyle()), Outp…

### Slider key

>Central mass density ($\rho_0$): The central mass density is the density at the center of the galaxy, changing this value changes the magnitude of the Dark Matter curve.

>Core radius ($r_c$): The core radius (also called "cutoff radius" or "scale radius") indicates where the density falls off by a factor of e (~2.7). Adjusting this factor changes where the "bump" of the curve is located.

>$kpc$ = kiloparsec, equal to 3,260 light years, or $1.917 \times 10^{16}$ miles

>$km/s$ = kilometers per second (velocity unit)

>$M_{\odot}$ = solar masses (X number of masses equal to our sun), approximately equal to $1.989 \times 10^{30}$ kilograms

>$M_{\odot} / kpc^3$ = solar masses per kiloparsec cubed (density unit = mass/volume)

### References:

>Carroll, B. W., & Ostlie, D. A. (2018). **An Introduction to Modern Astrophysics**. Cambridge University Press.

Jimenez, Raul, Licia Verde, and S. Peng Oh. **Dark halo properties from rotation curves.** _Monthly Notices of the Royal Astronomical Society_ 339, no. 1 (2003): 243-259. https://doi.org/10.1046/j.1365-8711.2003.06165.x.

>Noordermeer, E., & Van Der Hulst, J. M. (2007). **The stellar mass distribution in early-type disc galaxies: Surface Photometry and bulge–disc decompositions.** Monthly Notices of the Royal Astronomical Society, 376(4), 1480–1512. https://doi.org/10.1111/j.1365-2966.2007.11532.x

>Noordermeer, E. (2008), **The rotation curves of flattened Sérsic bulges**. Monthly Notices of the Royal Astronomical Society, 385: 1359-1364. https://doi.org/10.1111/j.1365-2966.2008.12837.x

>Taylor, John Robert. __An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements.__ 2nd ed. United States of America: University Science Books, 1997.


# OPTIMIZATION PHASES

### List of variables

            Variable
            Description
            Comment

            $B$
            Number of full blocks/pages that need the records
            $\lceil \frac{|T|}{R} \rceil$; $B \ll |T|$

            $R$
            Number of records per block/page

            $|T|$
            Cardinality. Number of tuples of a table
            Size of table

            $D$
            Time to access (read or write) a disk block
            Approximately 0'010 seconds

            $C$
            Time for the CPU to process a record
            Approximately 10-9

            $d$
            Tree order
            Usually greater than 100

            $h$
            Tree depth minus 1
            $\lceil \log_u |T| \rceil - 1$

            $v$
            Number of different values in a search

            $u$
            $\%load \cdot 2d$

            $k$
            Number of repetitions of every value in the search

            ndist(A)
            Number of different values for attribute A
            Obtained from DB catalog

            max
            Maximum value of an attribute
            Obtained from DB catalog

            min
            Minimum value of an attribute
            Obtained from DB catalog

            $H$
            Time to evaluate the hash function

            $M$
            Memory pages for a join/sorting algorithm

            bits
            Bits per index block

            Domain Cardinality
            Maximum number of different values

### List of variables for intermediate results

Record length. $\sum$ attribute lengthi (+ control information)
$$|R|$$

Number of records per block
$$R_R = \lfloor \frac{B}{|R|} \rfloor$$

Number of blocks per table
$$B_R = \lceil \frac{|R|}{R_R} \rceil$$

###  Cardinalities estimation

Selectivity Factor. % of tuples in the output regarding the input. ~0: very selective ~1: not very selective
$$\mathrm{SF}$$

Output of cardinality estimation
$$|O| = \mathrm{SF} \cdot |R|$$ or $$|O| = \mathrm{SF} \cdot |R1| \cdot |R2|$$

Selection
$$|\mathrm{selection}(R)| = \mathrm{SF} \cdot |R|$$

Join
$$|\mathrm{join}(R, S)| = \mathrm{SF} \cdot |R| \cdot |S|$$

Unions with repetitions
$$|\mathrm{union}(R, S)| = |R| + |S|$$

Unions without repetitions
$$|\mathrm{union}(R, S)| = |R| + |S| - |\mathrm{union}(R, S)|$$

Difference (anti-join)
$$|\mathrm{difference}(R, S)| = |R| - |\mathrm{union}(R, S)|$$

## Optimization phases

**Asumptions**

* Materialized views
* Focus on Disk access time
* Physical address of the record
* Only consider cases
  1. No index
  2. Unordered B-Tree with addresses (B+)
  3. Unordered Hash with addresses
  4. Orderered B-Tree with addresses (Clustered)

### Unordered B-tree with addresses (B+)

**Assumptions**

* In every tree node **2d** addresses fit
* Tree load 66% (2/3)

### Orderered B-Tree with addresses (Clustered)

**Assumptions**

* Tree load 66% (2/3) (index and table blocks)

### Unordered Hash with addresses

**Assumptions**

* No blocks for excess
* The same number of entries fit in a bucket block as in a tree block
* Bucket blocks at 80% (4/5)

## Space

**No index**

$$B$$

**B+**

$$\sum_1^{h+1} \lceil \frac{|T|}{u^i} \rceil + B$$

**Clustered**

$$\sum_1^{h+1} \lceil \frac{|T|}{u^i} \rceil + \lceil 1.5B \rceil$$

**Hash**

$$1 + \lceil 1.25(\frac{|T|}{2d}) \rceil + B$$

Example:

$$\mathrm{Lvl_1} = \frac{|T|}{u}$$

$$\mathrm{Lvl_2} = \frac{|T|}{u^2}$$

$$\mathrm{Lvl_3} = \frac{|T|}{u^3}$$

## Access paths

### Table scan

The whole table

 $u = \frac{2}{3} \cdot 2d$

**No index**

$$B \cdot D$$

**B+**

Only useful for sort

$$\lceil \frac{|T|}{u} \rceil \cdot D + |T| \cdot D$$

**Clustered**

$$\lceil 1.5B \rceil \cdot D$$

**Hash**

Useless

$$\lceil 1.25(\frac{|T|}{2d}) \rceil \cdot D + |T| \cdot D $$

### Search one tuple

Equality of unique attribute

    $u = \frac{2}{3} \cdot 2d$

    $h = \lceil \log_u |T| \rceil - 1$

**No index**

$$0.5B \cdot D$$

**B+**

$$h \cdot D + D$$

**Clustered**

$$h \cdot D + D$$

**Hash**

$$H + D + D$$

### Search several tuples

Interval

No unique attribute

    $u = \frac{2}{3} \cdot 2d$

    $h = \lceil \log_u |T| \rceil - 1$

    $|O|$: cardinality of Output

    $v$: value in range

    $k$: repetitions per value

**No index**

$$B \cdot D$$

**B+**

$$h \cdot D + \frac{|O| - 1}{u} \cdot D + |O| \cdot D$$

**Clustered**

$$h \cdot D + D + 1.5 \left( \frac{|O|-1}{R} \right) \cdot D$$

**Hash**

$$v = 1: 1 \cdot (H + D + k \cdot D) = H + D + k \cdot D$$

$$v > 1: v \cdot (H + D + k \cdot D)$$

$$v \;\mathrm{is\;unknown}: \mathrm{Useless}$$

### Statistics in Oracle

DBA is responsible for the statistics.

`ANALYZE [TABLE|INDEX|CLUSTER] <name> [COMPUTE|ESTIMATE] STATISTICS;`

```sql
ANALYZE TABLE departments COMPUTE STATISTICS;
ANALYZE TABLE employees COMPUTE STATISTICS;
```

`DBMS_STATS.GATHER_TABLE_STATS( <esquema>, <table> );`

```sql
DBMS_STATS.GATHER_TABLE_STATS("username", "departments");
DBMS_STATS.GATHER_TABLE_STATS("username", "employees");
```

Kinds of statistics

| Relations | Attributes |
|:--|:--|
| Cardinality | Length |
| Number of blocks | Domain cardinality |
| Average length of records | Number of existing different values |
| | Maximum value |
| | Minimum value |

Main hypothesis in most DBMS
* Uniform distribution of values for each attribute
* Independence of attributes

## Selectivity Factor of a Selection

Assuming equi-probability of values

`WHERE A = c`
$$\mathrm{SF}(A = c) = \frac{1}{\mathrm{ndist}(A)}$$

Assuming uniform distribution and $A \in [\min, \max]$

`WHERE A > c`

$$
\mathrm{SF}(A > c) = \frac{\max - c}{\max - \min} =
\begin{cases}
0 & \quad \text{if}\; c \geq \max \\
1 & \quad \text{if}\; c < \min
\end{cases}
$$

`WHERE A < c`

$$
\mathrm{SF}(A < c) = \frac{c - \min}{\max - \min} =
\begin{cases}
0 & \quad \text{if}\; c \leq \min \\
1 & \quad \text{if}\; c > \max
\end{cases}
$$

Assuming $\text{ndist}(A)$ is big enough

`WHERE A <= c`
$$
\mathrm{SF}(A \leq c) = \mathrm{SF}(A < c)
$$

`WHERE A >= c`
$$
\mathrm{SF}(A \geq c) = \mathrm{SF}(A > c)
$$

Assuming P and Q statistically **independent**

`WHERE P AND Q`
$$
\text{SF}(P \;\text{AND}\; Q) = \text{SF}(P) \cdot \text{SF}(Q)
$$

`WHERE P OR Q`
$$
\text{SF}(P \;\text{OR}\; Q) = \text{SF}(P) + \text{SF}(Q) - \text{SF}(P) \cdot \text{SF}(Q)
$$

`WHERE NOT P`
$$
\text{SF}(\text{NOT}\;P) = 1 - \text{SF}(P)
$$

`WHERE A IN (c1, c2, ... , cn)`
$$
\text{SF}(A \in (c_1, c_2, \dots, c_n)) = \min(1, \frac{n}{\mathrm{ndist}(A)})
$$

`WHERE A BETWEEN (c1, c2)`
$$
\text{SF}(c_1 \leq A \leq c_2) = \frac{\min(c_2, \max)-\max(c_1, \min)}{\max - \min}
$$

### Selectivity Factor of a Join

For $R[A \theta B]S$

Too difficult to approximate this general case. Usually, the required statistics are not available because it
would be too expensive to maintain them.

Results depend on operator:

$$
\text{SF}(R[A\times B]S) = 1
$$

$$
\text{SF}(R[A \neq B]S) = 1
$$

$$
\text{SF}(R[A=B]S) = \frac{1}{|R|}
\begin{cases}
S_B & \quad \text{is not null} \\
S_B & \quad \text{FK to } R_A \\
R_A & \quad \text{PK}
\end{cases}
$$

If there is no FK

$$
\text{SF}(R[A=B]S) = \frac{1}{\max(\text{ndist}(A), \text{ndist}(B))}
$$

$$
\text{SF}(R[A<B]S) = {^1/_2}
$$
$$
\text{SF}(R[A \leq B]S) = {^1/_2}
$$

## Phases of physiscal optimization

1. Alternatives generation
2. Intermediate results estimation
3. Cost estimation for each algorithm
4. Choose the best option

## Example

```sql
SELECT
    DISTINCT w.strength
FROM
    wines w, producers p, vintages v
WHERE
    v.wineId = w.wineId
    AND
    p.prodId = v.prodId
    AND
    p.region = "Priorat"
    AND
    v.quantity > 100;
```

Tables have the following structures

Producers
* Clustered by `prodId`
* B+ by `region`

Wines
* Clustered by `wineId`

Vintages
* Clustered by `wineId` and `prodId`

Statistics:
Tables (extra space due to being clustered needs to be added)

$$
\begin{matrix}
|P| = 10000 & |W| = 5000 & |V| = 100000 \\
R_p = 12    & R_w = 10 & R_v = 20 \\
B_p = 834   & B_w = 500  & B_v = 5000
\end{matrix}
$$

Attributes

prodId, wineId and strength: $|R_R| = 5$ bytes

$\text{ndist(region)} = 30$

$\min(\text{quantity}) = 10$ $\max(\text{quantity}) = 500$

$\text{ndist(strength)} = 10$

**System Parameters**

$B = 500$ bytes per intermediate disk block

$D = 1$

$C = 0$

$d = 75$

DBMS:

* Block Nested Loops (6 Memory pages, $M = 4$)
* Row Nested Loops
* Sort Match (with 3 memory pages for sorting, $M = 2$)

```python
import math

c_P, c_W, c_V = 10000, 5000, 100000
R_p, R_w, R_v = 12, 10, 20
B_p, B_w, B_v = math.ceil(c_P / R_p), math.ceil(c_W / R_w), math.ceil(c_V / R_v)

print("Cardinality of {}: {}, Records: {}, number of Full Blocks: {}".format('P', c_P, R_p, B_p))
print("Cardinality of {}: {}, Records: {}, number of Full Blocks: {}".format('W', c_W, R_w, B_w))
print("Cardinality of {}: {}, Records: {}, number of Full Blocks: {}".format('V', c_V, R_v, B_v))
```

    Cardinality of P: 10000, Records: 12, number of Full Blocks: 834
    Cardinality of W: 5000, Records: 10, number of Full Blocks: 500
    Cardinality of V: 100000, Records: 20, number of Full Blocks: 5000

### Phase 1. Alternatives generation

```sql
SELECT
    DISTINCT w.strength
FROM
    wines w, producers p, vintages v
WHERE
    v.wineId = w.wineId AND p.prodId = v.prodId
    AND p.region = "Priorat"
    AND v.quantity > 100;
```

Change selection and join arrangement

### Phase 2. Intermediate results estimation

```sql
SELECT
    DISTINCT w.strength
FROM
    wines w, producers p, vintages v
WHERE
    v.wineId = w.wineId AND p.prodId = v.prodId
    AND p.region = "Priorat"
    AND v.quantity > 100;
```

**PT1 and PT2**

**Selection over V: V'**

Record length of prodId and wineId:
$$|R_{V'}| = 5 + 5 = 10$$

Selectivity factor of selection:
$$
\mathrm{SF}(A > c) = \frac{\max - c}{\max - \min}
$$

Where $c = 100$ and the query specifies `v.quantity > 100`, then:

$$
\text{SF}(\text{quantity} > 100) = \frac{500 - 100}{500 - 10} = 0.81632
$$

Output cardinality of V':
$$
|O| = \text{SF} \cdot |R|
$$

$$
|V'| = \text{SF}(\text{quantity} > 100) \cdot |V| = 0.81632 \cdot 100000 = 81632
$$

Number of records per block:
$$
R_{V'} = \lfloor \frac{B}{|R_{V'}|} \rfloor = \lfloor \frac{500}{10} \rfloor = 50
$$

Number of blocks needed for V':
$$
B_{V'} = \lceil \frac{|V'|}{R_{V'}} \rceil = \lceil \frac{81632}{50} \rceil = 1633
$$

```python
c = 100
min_v = 10
max_v = 500
SF_v_prime = (max_v - c) / (max_v - min_v)
print("Selectivity factor of V': {} \n".format(SF_v_prime))

C_v_prime = math.floor(SF_v_prime * c_V)
print("Cardinality output of V': {} \n".format(C_v_prime))

R_v_len = 5 + 5
B = 500
R_v_prime = math.floor(B / R_v_len)
print("V' number of records per block : {} \n".format(R_v_prime))

B_v_prime = math.ceil(C_v_prime / R_v_prime)
print("Blocks needed for V': {} \n".format(B_v_prime))
```

    Selectivity factor of V': 0.8163265306122449

    Cardinality output of V': 81632

    V' number of records per block : 50

    Blocks needed for V': 1633

**Selection over P: P'**

Record length of prodId:
$$|R_{P'}| = 5$$

Selectivity factor of selection:
$$
\mathrm{SF}(A = c) = \frac{1}{\text{ndist}(A)}
$$

Where $c = 'Priorat'$ and the query specifies `p.region = 'Priorat'`, then:

$$
\text{SF}(\text{region} = \text{Priorat}) = \frac{1}{30} = 0.033333
$$

Output cardinality of P':
$$
|O| = \text{SF} \cdot |R|
$$

$$
|P'| = \text{SF}(\text{region} = \text{Priorat}) \cdot |P| = 0.03333 \cdot 10000 = 333
$$

Number of records per block:
$$
R_{P'} = \lfloor \frac{B}{|R_{P'}|} \rfloor = \lfloor \frac{500}{5} \rfloor = 100
$$

Number of blocks needed for P':
$$
B_{P'} = \lceil \frac{|P'|}{R_{P'}} \rceil = \lceil \frac{333}{100} \rceil = 4
$$

```python
ndist_region = 30
SF_p_prime = 1 / ndist_region
print("Selectivity factor of P': {} \n".format(SF_p_prime))

C_p_prime = math.floor(SF_p_prime * c_P)
print("Cardinality output of P': {} \n".format(C_p_prime))

R_p_len = 5
B = 500
R_p_prime = math.floor(B / R_p_len)
print("P' number of records per block : {} \n".format(R_p_prime))

B_p_prime = math.ceil(C_p_prime / R_p_prime)
print("Blocks needed for P': {} \n".format(B_p_prime))
```

    Selectivity factor of P': 0.03333333333333333

    Cardinality output of P': 333

    P' number of records per block : 100

    Blocks needed for P': 4

**PT1**

**Join between W and V': WV'**

Record length of `strength` and `prodId`:
$$
|R_{WV'}| = 5 + 5
$$

Selectivity factor

$$
\text{SF}_{WV'} = \frac{1}{|W|} = \frac{1}{5000} = 0.0002
$$

Cardinality ouput of WV'
$$
|WV'| = SF_{WV'} \cdot |W| \cdot |V'| = \frac{1}{5000} \cdot 5000 \cdot  81632 = 81632
$$

Number of rows per block for WV':
$$
R_{WV'} = \lfloor \frac{B}{|R_{WV'}|} \rfloor = \lfloor \frac{500}{5} \rfloor = 50
$$

Number of blocks used for WV':

$$
B_{WV'} = \lceil \frac{|WV'|}{R_{WV'}} \rceil = \lceil \frac{81632}{50} \rceil = 1633
$$

```python
SF_wv_prime = 1 / c_W
print("Selectivity factor of WV': {} \n".format(SF_wv_prime))

C_wv_prime = math.floor(SF_wv_prime * c_W * C_v_prime)
print("Cardinality output of WV': {} \n".format(C_wv_prime))

R_wv_prime_len = 5 + 5
B = 500
R_wv_prime = math.floor(B / R_wv_prime_len)
print("WV' number of records per block : {} \n".format(R_wv_prime))

B_wv_prime = math.ceil(C_wv_prime / R_wv_prime)
print("Blocks needed for WV': {} \n".format(B_wv_prime))
```

    Selectivity factor of WV': 0.0002

    Cardinality output of WV': 81632

    WV' number of records per block : 50

    Blocks needed for WV': 1633

**Join between WV' and P': WV'P'**

Record length for `strength`:
$$
|R_{WV'P'}| = 5
$$

Selectivity Factor, assuming quantity and region independent
$$
\text{SF(WV'} \cdot \text{P')} = \frac{1}{|P'|} \cdot \frac{1}{ndist(\text{region})} = \frac{1}{333 \cdot 30} = 10^{-4}
$$

Cardinality output
$$
|WV'P'| = SF_{WV'P'} \cdot |WV'| \cdot |P'| = 10^{-4} \cdot 81632 \cdot 333 = 2721
$$

Records per block
$$
R_{WV'P'} = \lfloor \frac{B}{|R_{WV'P'}|} \rfloor = \lfloor \frac{500}{5} \rfloor = 100
$$

Blocks for WV'P'
$$
B_{WV'P'} = \lceil \frac{|WV'P'|}{R_{WV'P'}} \rceil = \lceil \frac{1234}{100} \rceil = 28
$$

```python
SF_wvp_prime = (1 / C_p_prime) * (1 / ndist_region)
print("Selectivity factor of WV'P': {} \n".format(SF_wvp_prime))

C_wvp_prime = math.floor(SF_wvp_prime * C_wv_prime * C_p_prime)
print("Cardinality output of WV'P': {} \n".format(C_wvp_prime))

R_wvp_prime_len = 5
B = 500
R_wvp_prime = math.floor(B / R_wvp_prime_len)
print("WV'P' number of records per block : {} \n".format(R_wvp_prime))

B_wvp_prime = math.ceil(C_wvp_prime / R_wvp_prime)
print("Blocks needed for WV'P': {} \n".format(B_wvp_prime))
```

    Selectivity factor of WV'P': 0.00010010010010010009

    Cardinality output of WV'P': 2721

    WV'P' number of records per block : 100

    Blocks needed for WV'P': 28

**PT2**

**Join V' and P': V'P'**

Assuming independence of variables

Record length for `wineId`
$$
|R_{V'P'}| = 5
$$

Selectivity factor
$$
\text{SF}_{V'P'} = \frac{1}{ndist(\text{region})} \cdot \frac{1}{|P'|} = \frac{1}{30} \cdot \frac{1}{333} = 10^{-4}
$$

Output cardinality
$$
|V'P'| = \text{SF}_{V'P'} \cdot |V'| \cdot |P'| = 10^{-4} \cdot 81632 \cdot 333 = 2721
$$

Number of records per blocks
$$
R_{V'P'} = \lfloor \frac{B}{R_{V'P'}} \rfloor = \lfloor \frac{500}{R_{5}} \rfloor = 100
$$

Blocks needed for V'P'
$$
B_{V'P'} = \lceil \frac{|V'P'|}{R_{V'P'}} \rceil = \lceil \frac{2721}{100} \rceil = 28
$$

```python
ndist_region = 30
SF_vp_prime = (1 / ndist_region) * (1 / C_p_prime)
print("Selectivity factor of V'P': {} \n".format(SF_vp_prime))

C_vp_prime = math.floor(SF_vp_prime * C_v_prime * C_p_prime)
print("Cardinality output of V'P': {} \n".format(C_vp_prime))

R_vp_len = 5
B = 500
R_vp_prime = math.floor(B / R_vp_len)
print("V'P' number of records per block : {} \n".format(R_vp_prime))

B_vp_prime = math.ceil(C_vp_prime / R_vp_prime)
print("Blocks needed for V'P': {} \n".format(B_vp_prime))
```

    Selectivity factor of V'P': 0.00010010010010010009

    Cardinality output of V'P': 2721

    V'P' number of records per block : 100

    Blocks needed for V'P': 28

**Join between W and V'P': WV'P'**

Record length for WV'P'
$$
|R_{WV'P'}| = 5
$$

Selectivity Factor for WV'P'
$$
\text{SF} = \frac{1}{|W|} = \frac{1}{5000} = 0.0002
$$

Cardinality Output
$$
|WV'P'| = SF \cdot |W| \cdot |V'P'| = 10^{-4} \cdot 5000 \cdot 2721 = 2721
$$

Number of records per block
$$
R_{WV'P'} = \lfloor \frac{B}{|R_{WV'P'}|} \rfloor = \lfloor \frac{500}{5} \rfloor = 100
$$

Blocks needes for WV'P'
$$
B_{WV'P'} = \lceil \frac{|WV'P'|}{R_{WV'P'}} \rceil = \lceil \frac{2721}{100} \rceil = 28
$$

```python
SF_wv_pr_p_pr = 1 / c_W
print("Selectivity factor of WV'P': {} \n".format(SF_wv_pr_p_pr))

C_wv_pr_p_pr = math.floor(SF_wv_pr_p_pr * c_W * C_vp_prime)
print("Cardinality output of WV'P': {} \n".format(C_wv_pr_p_pr))

R_wv_pr_p_pr_len = 5
B = 500
R_wv_pr_p_pr = math.floor(B / R_wv_pr_p_pr_len)
print("WV'P' number of records per block : {} \n".format(R_wv_pr_p_pr))

B_wv_pr_p_pr = math.ceil(C_wv_pr_p_pr / R_wv_pr_p_pr)
print("Blocks needed for WV'P': {} \n".format(B_wv_pr_p_pr))
```

    Selectivity factor of WV'P': 0.0002

    Cardinality output of WV'P': 2721

    WV'P' number of records per block : 100

    Blocks needed for WV'P': 28

**PT1/PT2**

**Final result = O**

Record length
$$
|R_O| = 5
$$

Output cardinality
$$
|O| = \text{ndist}(\text{strength}) = 100
$$

Number of records
$$
R_O = \lfloor \frac{B}{|R_O|} \rfloor = \lfloor \frac{500}{5} \rfloor = 100
$$

Blocks needed
$$
B_O = \lceil \frac{|O|}{R_O} \rceil = \lceil \frac{100}{100} \rceil = 1
$$

```python
ndist_strength = 100
C_o = ndist_strength
print("Cardinality output of O: {} \n".format(C_o))

R_o_len = 5
B = 500
R_o = math.floor(B / R_o_len)
print("O number of records per block : {} \n".format(R_o))

B_o = math.ceil(C_o / R_o)
print("Blocks needed for O: {} \n".format(B_o))
```

    Cardinality output of O: 100

    O number of records per block : 100

    Blocks needed for O: 1

**Map result**

### Phase 3. Cost estimation for each algorithm

Recall:

$$
u = \frac{2}{3} \cdot 2(75) = 100
$$

**AP1/AP2**

**Selection over V: V'**

Recall that for Vintages is clustered by wineId and prodId
Available access paths: No index

$$
\text{cost}_{\text{scan}}(V') = \lceil 1.5 B_{V} \rceil \cdot D = \lceil 1.5 \cdot 5000 \rceil \cdot 1 = 7500
$$

Chosen algorithm: **Scan**

**Selection over P: P'**

Available access paths: B+ and no index

For a table scan
$$
\text{cost}_{\text{scan}}(P') = \lceil 1.5 B_{P} \rceil \cdot D = \lceil 1.5 \cdot 834 \rceil \cdot 1 = 1251
$$

Tree depth of h for B+ is:

$$
h = \lceil \log_u |P| \rceil - 1 =  \lceil \log_u |P| \rceil - 1 =  \lceil \log_{100} 10000 \rceil - 1 = 1
$$

For an index of several tuples
$$
\begin{align}
\text{cost}_{B^+}(P')
& = h \cdot D + \frac{|P'| - 1}{u} \cdot D + |P'| \cdot D \\
& = h \cdot D + \frac{SF_{\text{region = 'Priorat'}} \cdot |P| - 1}{u} \cdot D + SF_{\text{region = 'Priorat'}} \cdot |P| \cdot D \\
& = 1 \cdot 1 + \frac{{^1/_{30}} \cdot 10000 - 1}{100} \cdot D + {^1/_{30}} \cdot 10000 \cdot 1 \\
& = 1 + \frac{332}{100} + 333 \\
& = 337.33
\end{align}
$$

Chosen algorithm: **B+**

```python
load = 2/3
d = 75
u = load * (2 * d)
h = math.ceil(math.log(c_P, u)) - 1
D = 1
print("load is: {}\nd is: {}\nu is: {}\nh is: {}\nD is: {}\n".format(load, d, u, h, D))
cost_scan_p = math.ceil(1.5 * B_p) * D
cost_bplus_p = (h * D) + ((C_p_prime / u) * D) + (C_p_prime * D)
print("Cost of scan is: {} \nCost of B+ is: {}".format(cost_scan_p, cost_bplus_p))
```

    load is: 0.6666666666666666
    d is: 75
    u is: 100.0
    h is: 1
    D is: 1

    Cost of scan is: 1251
    Cost of B+ is: 337.33

**PT1**

**Join over W and V': WV'**

Available algorithms: Block Nested Loops (BML), Row Nested Loops (RML) and Sort-Match (SM)

*Block Nested Loops*

Recall:

$$
M = 4
$$

$\lceil 1.5 B_{W} \rceil < B_{V'}$ we use the commutative property of joins

$$
\begin{align}
\text{cost}_{\text{BML}}(WV')
& = \lceil 1.5 B_{W} \rceil + \lceil \frac{1.5 B_{W}}{M} \rceil \cdot B_{V'} \\
& = \lceil 1.5 \cdot 500 \rceil + \lceil \frac{1.5\cdot 500}{4} \rceil \cdot 1633 \\
& = 307,754
\end{align}
$$

*Row Nested Loops*

Look for attributes of W

$V'$ does not use extra space any more for being ordered

$$
\begin{align}
\text{cost}_{\text{RML}}(WV')
& = B_{V'} + |V'| \cdot \left( \lceil \log_u |W| \rceil - 1 + 1 + (\frac{1.5(k-1)}{10} \right) \\
& = 1633 + 81,632 \cdot \left( \lceil \log_{100} 5000 \rceil - 1 + 1 \right) \\
& = 164,887
\end{align}
$$

Note: This wasn't explained. Maybe $k = 1$ but needs confirmation.

*Sort-Match*

$W$ is ordered by `wineID`, $V'$ is still ordered y `wineID` and `prodID`.

$$
\text{cost}_{\text{SM}}(WV') = \lceil 1.5 B_{W} \rceil + B_{V'} = \lceil 1.5 \cdot 500 \rceil + 1633 = 2383
$$

Chosen algorithm: **Sort-Match**

**Join between WV' and P': WV'P'**

*Block Nested Loops*

$B_{P'} < B_{WV'}$ we use the commutative property of joins

$$
\begin{align}
\text{cost}_{\text{BML}}(WV'P')
& = B_{P'} + \lceil \frac{B_{P'}}{M} \rceil \cdot B_{WV'} \\
& = 4 + \lceil \frac{4}{4} \rceil \cdot 1633 \\
& = 1637
\end{align}
$$

Note: It isn't explained why BML is analyzed but not RML.

*Sort-Match*

Neither WV’ nor P’ are ordered by `prodID`

$$
\begin{align}
\text{cost}_{\text{SM}}(WV'P')
&= 2 B_{WV'} \cdot \lceil \log_2 B_{WV'} \rceil + 2 B_{P'} \cdot \lceil \log_2 B_{P'} \rceil + B_{WV'} + B_{P'}\\
&= 2 \cdot 1633 \cdot \lceil \log_2 1633 \rceil + 2 \cdot 4 \cdot \lceil \log_2 4 \rceil + 1633 + 4 \\
&= 37,579
\end{align}
$$

Chosen algorithm: **Block Nested Loop**

```python
print("B_p' is {}\nB_wv' is {}".format(B_p_prime, B_wv_prime))
```

    B_p' is 4
    B_wv' is 1633

```python
(2 * B_wv_prime * math.ceil(math.log(B_wv_prime, 2))) + (2 * B_p_prime * math.ceil(math.log(B_p_prime, 2))) + B_wv_prime + B_p_prime
```

    37579

**PT2**

**Join between V' and P': V'P'

Available algorithms: BNL and SM.

*Block Nested Loops*

$B_{P'} < B_{V'}$ we use the commutative property of joins

$$
\begin{align}
\text{cost}_{\text{BML}}(V'P')
& = B_{P'} + \lceil \frac{B_{P'}}{M} \rceil \cdot B_{V'} \\
& = 4 + \lceil \frac{4}{4} \rceil \cdot 1633 \\
& = 1637
\end{align}
$$

*Sort-Match*

Neither V’ nor P’ are ordered by `prodID`

$$
\begin{align}
\text{cost}_{\text{SM}}(V'P')
&= 2 B_{V'} \cdot \lceil \log_2 B_{V'} \rceil + 2 B_{P'} \cdot \lceil \log_2 B_{P'} \rceil + B_{V'} + B_{P'}\\
&= 2 \cdot 1633 \cdot \lceil \log_2 1633 \rceil + 2 \cdot 4 \cdot \lceil \log_2 4 \rceil + 1633 + 4 \\
&= 37,579
\end{align}
$$

Chosen algorithm: **Block Nested Loop**

```python
print("B_p' is {}\nB_v' is {}".format(B_p_prime, B_v_prime))
```

    B_p' is 4
    B_v' is 1633

**Join between W and V'P': WV'P'**

Available algorithms: Block Nested Loops (BML), Row Nested Loops (RML) and Sort-Match (SM)

*Block Nested Loops*

$B_{V'P'} < \lceil 1.5 B_{W} \rceil$ we use the commutative property of joins

$$
\begin{align}
\text{cost}_{\text{BML}}(WV'P')
& = B_{V'P'} + \lceil \frac{B_{V'P'}}{M} \rceil \cdot \lceil 1.5 B_{W} \rceil \\
& = 28 + \lceil \frac{28}{4} \rceil \cdot \lceil 750 \rceil \\
& = 5278
\end{align}
$$

*Row Nested Loops*

Look for attributes of W

$$
\begin{align}
\text{cost}_{\text{RML}}(WV'P')
& = B_{V'P'} + |V'P'| \cdot \left( \lceil \log_u |W| \rceil - 1 + 1 + (\frac{1.5(k-1)}{10} \right) \\
& = 28 + 2721 \cdot \left( \lceil \log_{100} 5000 \rceil - 1 + 1 \right) \\
& = 5470
\end{align}
$$

*Sort-Match*

W is ordered by `wineID`, V'P' is not sorted by `wineID`

$$
\begin{align}
\text{cost}_{\text{SM}}(WV'P')
&= 2 B_{V'P'} \cdot \lceil \log_2 B_{V'P'} \rceil + \lceil 1.5 B_{W} \rceil + B_{V'P'} \\
&= 2 \cdot 28 \cdot \lceil \log_2 28 \rceil + \lceil 1.5 \cdot 500 \rceil + 28 \\
&= 1058
\end{align}
$$

Chosen algorithm: **Sort-Match**

```python
print("B_v'p' is {}\n1.5*B_w is {}\n|V'P'| is {}".format(B_vp_prime, math.ceil(1.5*B_w), C_vp_prime))
```

    B_v'p' is 28
    1.5*B_w is 750
    |V'P'| is 2721

```python
28 + math.ceil(28/4) * 750
```

    5278

```python
28+(2721*(math.ceil(math.log(5000, 100)) - 1 + 1))
```

    5060.448690899154

```python
Cost_v_prime = 1633 + 7500
Cost_p_prime = 4 + 337
Cost_wv = 1633 + 2383
Cost_vp = 28 + 1637
Cost_wvp_pt1 = 28 + 1637
Cost_wvp_pt2 = 28 + 1058
Cost_o = 1 + 252
Cost_pt1 = Cost_v_prime + Cost_p_prime + Cost_wv + Cost_wvp_pt1 + Cost_o
Cost_pt2 = Cost_v_prime + Cost_p_prime + Cost_vp + Cost_wvp_pt2 + Cost_o
print("Total cost of:\nPT1: {}\nPT2: {}".format(Cost_pt1, Cost_pt2))
```

    Total cost of:
    PT1: 15408
    PT2: 12478

**Map result**

Output algorithm is Merge Sort but it's not explained, nor its cost calculation

### Phase 4. Choose the best option

**PT2**

```python

```


# Deep Learning 101

This notebook presents the basics concepts that involve the concept of Deep Learning.

1. Linear Regression
* Logistic Regression
* **Artificial Neural Networks**
* Deep Neural Networks
* Convolutional Neural Networks

## 3. Artificial Neural Networks

These models are called *feedforward* because information **flows** through the function being evaluated from $x$, through the intermediate computations used to define $f$, and finally to the output $y$.

*Feedforward* neural networks are called networks because they are typically represented by composing together many different functions.

$$
    \begin{align}
    \hat{y} &= f(x) \\
            &= f^{(2)}(f^{(1)}(x))
    \end{align}
$$

$$f^{(i)}(h) = \sigma_i(W_ih + b_i)$$

$$\hat{y} = \sigma_2(W_2 \sigma_1(W_1 x + b_1) + b_2)$$
---

## Artificial Neural Networks with Keras and TensorFlow

**Activations functions** are non-linear functions applied when passing the output of a layer to the next layer or the final

| **Keras**      | **Example**                                                         | **Link**                 |
| -------------- |:------------------------------------------------------------------: | :-----------------------:|
| **Activation functions**| sigmoid, step, tanh, softmax, ReLU | [keras.io/activations/](https://keras.io/activations/)|

---

## 1. Load data

#### Load libraries

```python
%matplotlib inline
import matplotlib.pyplot as plt

import numpy as np
```

```python
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import plot_model, to_categorical
```

#### Getting the data

```python
# load dataset
(X_train, y_train), (X_test, y_test) = mnist.load_data()
```

#### Explore visual data

```python
fig = plt.figure()
for i in range(10):
    plt.subplot(2, 5, i+1)
    x_y = X_train[y_train == i]
    plt.imshow(x_y[0], cmap='gray', interpolation='none')
    plt.title("Class %d" % (i))
    plt.xticks([])
    plt.yticks([])

plt.tight_layout()
```

```python
print('X_train.shape', X_train.shape)
print('y_train.shape', y_train.shape)
print('X_test.shape', X_test.shape)
print('y_train.shape', y_test.shape)
```

    X_train.shape (60000, 28, 28)
    y_train.shape (60000,)
    X_test.shape (10000, 28, 28)
    y_train.shape (10000,)

#### Reshaping and normalizing the inputs

```python
# reshaping the inputs
X_train = X_train.reshape(60000, 28*28)
X_test = X_test.reshape(10000, 28*28)

# normalizing the inputs
X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0
print('X_train reshape:', X_train.shape)
print('X_test reshape:', X_test.shape)
```

    X_train reshape: (60000, 784)
    X_test reshape: (10000, 784)

#### Convert class vectors to binary class matrices

```python
# 10 classes
y_train_cat = to_categorical(y_train, 10)
y_test_cat = to_categorical(y_test, 10)

print('y_train_cat shape:', y_train_cat.shape)
print('y_test_cat shape:', y_test_cat.shape)
```

    y_train_cat shape: (60000, 10)
    y_test_cat shape: (10000, 10)

## 2. Define model

#### Add the input-, hidden- and output-layers

```python
# building a linear stack of layers with the sequential model
model = Sequential()

# Add the input layer and hidden layer 1
model.add(Dense(32, input_shape=(784,), activation='sigmoid'))

# Add the output layer
model.add(Dense(10, activation='softmax'))
```

    2021-08-08 20:22:18.077419: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

#### Model visualization

```python
# plot a Keras model
plot_model(model, to_file='img/model03_ann.png',
           show_shapes=True, show_layer_names=True)

# prints a summary representation of your model
model.summary()
```

    ('Failed to import pydot. You must `pip install pydot` and install graphviz (https://graphviz.gitlab.io/download/), ', 'for `pydotprint` to work.')
    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense (Dense)                (None, 32)                25120
    _________________________________________________________________
    dense_1 (Dense)              (None, 10)                330
    =================================================================
    Total params: 25,450
    Trainable params: 25,450
    Non-trainable params: 0
    _________________________________________________________________

## 3. Compile model

```python
# compiling the sequential model
model.compile('rmsprop', loss='categorical_crossentropy',
              metrics=['categorical_accuracy'])
```

## 4. Fit model

```python
# training the model and saving metrics in history
history = model.fit(X_train, y_train_cat,
                    batch_size=256, epochs=50,
                    verbose=2,
                    validation_data=(X_test, y_test_cat))
```

    2021-08-08 20:22:18.239332: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
    2021-08-08 20:22:18.257943: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 1800000000 Hz

    Epoch 1/50
    235/235 - 1s - loss: 1.1466 - categorical_accuracy: 0.7612 - val_loss: 0.6066 - val_categorical_accuracy: 0.8823
    Epoch 2/50
    235/235 - 0s - loss: 0.4703 - categorical_accuracy: 0.8931 - val_loss: 0.3661 - val_categorical_accuracy: 0.9092
    Epoch 3/50
    235/235 - 0s - loss: 0.3327 - categorical_accuracy: 0.9122 - val_loss: 0.2948 - val_categorical_accuracy: 0.9195
    Epoch 4/50
    235/235 - 0s - loss: 0.2804 - categorical_accuracy: 0.9229 - val_loss: 0.2610 - val_categorical_accuracy: 0.9287
    Epoch 5/50
    235/235 - 0s - loss: 0.2507 - categorical_accuracy: 0.9301 - val_loss: 0.2393 - val_categorical_accuracy: 0.9321
    Epoch 6/50
    235/235 - 0s - loss: 0.2301 - categorical_accuracy: 0.9359 - val_loss: 0.2220 - val_categorical_accuracy: 0.9363
    Epoch 7/50
    235/235 - 0s - loss: 0.2136 - categorical_accuracy: 0.9400 - val_loss: 0.2093 - val_categorical_accuracy: 0.9393
    Epoch 8/50
    235/235 - 0s - loss: 0.2004 - categorical_accuracy: 0.9432 - val_loss: 0.1989 - val_categorical_accuracy: 0.9414
    Epoch 9/50
    235/235 - 0s - loss: 0.1892 - categorical_accuracy: 0.9464 - val_loss: 0.1923 - val_categorical_accuracy: 0.9430
    Epoch 10/50
    235/235 - 1s - loss: 0.1796 - categorical_accuracy: 0.9496 - val_loss: 0.1833 - val_categorical_accuracy: 0.9460
    Epoch 11/50
    235/235 - 0s - loss: 0.1711 - categorical_accuracy: 0.9518 - val_loss: 0.1776 - val_categorical_accuracy: 0.9480
    Epoch 12/50
    235/235 - 0s - loss: 0.1636 - categorical_accuracy: 0.9535 - val_loss: 0.1716 - val_categorical_accuracy: 0.9490
    Epoch 13/50
    235/235 - 0s - loss: 0.1568 - categorical_accuracy: 0.9557 - val_loss: 0.1666 - val_categorical_accuracy: 0.9510
    Epoch 14/50
    235/235 - 0s - loss: 0.1509 - categorical_accuracy: 0.9572 - val_loss: 0.1637 - val_categorical_accuracy: 0.9508
    Epoch 15/50
    235/235 - 0s - loss: 0.1455 - categorical_accuracy: 0.9592 - val_loss: 0.1588 - val_categorical_accuracy: 0.9523
    Epoch 16/50
    235/235 - 0s - loss: 0.1406 - categorical_accuracy: 0.9598 - val_loss: 0.1542 - val_categorical_accuracy: 0.9536
    Epoch 17/50
    235/235 - 0s - loss: 0.1363 - categorical_accuracy: 0.9613 - val_loss: 0.1519 - val_categorical_accuracy: 0.9543
    Epoch 18/50
    235/235 - 0s - loss: 0.1322 - categorical_accuracy: 0.9622 - val_loss: 0.1496 - val_categorical_accuracy: 0.9571
    Epoch 19/50
    235/235 - 0s - loss: 0.1282 - categorical_accuracy: 0.9634 - val_loss: 0.1467 - val_categorical_accuracy: 0.9556
    Epoch 20/50
    235/235 - 0s - loss: 0.1246 - categorical_accuracy: 0.9649 - val_loss: 0.1451 - val_categorical_accuracy: 0.9558
    Epoch 21/50
    235/235 - 0s - loss: 0.1212 - categorical_accuracy: 0.9658 - val_loss: 0.1436 - val_categorical_accuracy: 0.9582
    Epoch 22/50
    235/235 - 0s - loss: 0.1182 - categorical_accuracy: 0.9669 - val_loss: 0.1408 - val_categorical_accuracy: 0.9578
    Epoch 23/50
    235/235 - 1s - loss: 0.1152 - categorical_accuracy: 0.9674 - val_loss: 0.1392 - val_categorical_accuracy: 0.9580
    Epoch 24/50
    235/235 - 1s - loss: 0.1125 - categorical_accuracy: 0.9682 - val_loss: 0.1371 - val_categorical_accuracy: 0.9588
    Epoch 25/50
    235/235 - 0s - loss: 0.1096 - categorical_accuracy: 0.9691 - val_loss: 0.1361 - val_categorical_accuracy: 0.9597
    Epoch 26/50
    235/235 - 0s - loss: 0.1073 - categorical_accuracy: 0.9697 - val_loss: 0.1329 - val_categorical_accuracy: 0.9607
    Epoch 27/50
    235/235 - 0s - loss: 0.1050 - categorical_accuracy: 0.9704 - val_loss: 0.1320 - val_categorical_accuracy: 0.9623
    Epoch 28/50
    235/235 - 0s - loss: 0.1027 - categorical_accuracy: 0.9713 - val_loss: 0.1312 - val_categorical_accuracy: 0.9611
    Epoch 29/50
    235/235 - 0s - loss: 0.1004 - categorical_accuracy: 0.9715 - val_loss: 0.1312 - val_categorical_accuracy: 0.9614
    Epoch 30/50
    235/235 - 0s - loss: 0.0985 - categorical_accuracy: 0.9724 - val_loss: 0.1282 - val_categorical_accuracy: 0.9621
    Epoch 31/50
    235/235 - 0s - loss: 0.0962 - categorical_accuracy: 0.9728 - val_loss: 0.1274 - val_categorical_accuracy: 0.9631
    Epoch 32/50
    235/235 - 0s - loss: 0.0946 - categorical_accuracy: 0.9732 - val_loss: 0.1264 - val_categorical_accuracy: 0.9630
    Epoch 33/50
    235/235 - 0s - loss: 0.0929 - categorical_accuracy: 0.9738 - val_loss: 0.1264 - val_categorical_accuracy: 0.9625
    Epoch 34/50
    235/235 - 0s - loss: 0.0911 - categorical_accuracy: 0.9741 - val_loss: 0.1259 - val_categorical_accuracy: 0.9626
    Epoch 35/50
    235/235 - 1s - loss: 0.0893 - categorical_accuracy: 0.9746 - val_loss: 0.1253 - val_categorical_accuracy: 0.9625
    Epoch 36/50
    235/235 - 0s - loss: 0.0880 - categorical_accuracy: 0.9753 - val_loss: 0.1241 - val_categorical_accuracy: 0.9637
    Epoch 37/50
    235/235 - 1s - loss: 0.0865 - categorical_accuracy: 0.9754 - val_loss: 0.1240 - val_categorical_accuracy: 0.9630
    Epoch 38/50
    235/235 - 1s - loss: 0.0848 - categorical_accuracy: 0.9758 - val_loss: 0.1238 - val_categorical_accuracy: 0.9647
    Epoch 39/50
    235/235 - 1s - loss: 0.0833 - categorical_accuracy: 0.9762 - val_loss: 0.1230 - val_categorical_accuracy: 0.9651
    Epoch 40/50
    235/235 - 1s - loss: 0.0822 - categorical_accuracy: 0.9769 - val_loss: 0.1230 - val_categorical_accuracy: 0.9634
    Epoch 41/50
    235/235 - 1s - loss: 0.0807 - categorical_accuracy: 0.9776 - val_loss: 0.1215 - val_categorical_accuracy: 0.9644
    Epoch 42/50
    235/235 - 0s - loss: 0.0795 - categorical_accuracy: 0.9775 - val_loss: 0.1214 - val_categorical_accuracy: 0.9641
    Epoch 43/50
    235/235 - 0s - loss: 0.0782 - categorical_accuracy: 0.9779 - val_loss: 0.1230 - val_categorical_accuracy: 0.9633
    Epoch 44/50
    235/235 - 0s - loss: 0.0771 - categorical_accuracy: 0.9786 - val_loss: 0.1223 - val_categorical_accuracy: 0.9654
    Epoch 45/50
    235/235 - 0s - loss: 0.0759 - categorical_accuracy: 0.9785 - val_loss: 0.1215 - val_categorical_accuracy: 0.9646
    Epoch 46/50
    235/235 - 0s - loss: 0.0747 - categorical_accuracy: 0.9791 - val_loss: 0.1209 - val_categorical_accuracy: 0.9652
    Epoch 47/50
    235/235 - 0s - loss: 0.0735 - categorical_accuracy: 0.9796 - val_loss: 0.1208 - val_categorical_accuracy: 0.9656
    Epoch 48/50
    235/235 - 0s - loss: 0.0725 - categorical_accuracy: 0.9797 - val_loss: 0.1206 - val_categorical_accuracy: 0.9658
    Epoch 49/50
    235/235 - 0s - loss: 0.0716 - categorical_accuracy: 0.9798 - val_loss: 0.1200 - val_categorical_accuracy: 0.9646
    Epoch 50/50
    235/235 - 0s - loss: 0.0703 - categorical_accuracy: 0.9806 - val_loss: 0.1202 - val_categorical_accuracy: 0.9641

## 5. Evaluate model

```python
# plotting the metrics
fig = plt.figure()
plt.subplot(2,1,1)
plt.plot(history.history['categorical_accuracy'])
plt.plot(history.history['val_categorical_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='lower right')

plt.subplot(2,1,2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')

plt.tight_layout()
```

```python
# evaluate model on test data
[test_loss, test_acc] = model.evaluate(X_test, y_test_cat)
print("Evaluation result on Test Data:\nLoss = {}\nAccuracy = {}".format(test_loss, test_acc))
```

    313/313 [==============================] - 0s 512us/step - loss: 0.1202 - categorical_accuracy: 0.9641
    Evaluation result on Test Data:
    Loss = 0.12018612027168274
    Accuracy = 0.9641000032424927

## References

* [Deep Learning Book](http://www.deeplearningbook.org)

* [Zero to Deep Learning™ Udemy Video Course](https://github.com/dataweekends/zero_to_deep_learning_udemy)

* [THE MNIST DATABASE](http://yann.lecun.com/exdb/mnist/)