### Code setup

```python
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
sns.set_context("talk", font_scale=1.5, rc={"lines.linewidth": 2.5})
sns.set_style("whitegrid")
from IPython.display import HTML
from matplotlib import animation
%matplotlib inline

# Don't tinker, or do
#%matplotlib nbagg
# from matplotlib import rcParams
#rcParams['font.family']='sans-serif'
#rcParams('font', serif='Helvetica Neue')
# rcParams['text.usetex']= True
#rcParams.update({'font.size': 22})
```

## Solving initial (-boundary) value problems
Time is something we all have to deal with and manage. Within the domain of scientific applications, this usually defaults to solving a (system of) initial value problem, specified in the following form:
$$ \frac{\partial \mathbf{u}}{\partial t} = \mathbf{F}(\mathbf{u}, t)$$
as a partial differential equation (PDE). As we will see, on numerical discretization modifies the above PDE into a manageable (computable) form, wherein
$$ \mathbf{u}^{n+1} = \sum_{i=0}^{k} \alpha_i \mathbf{u}^{n-i} + \sum_{j=0}^{r} \beta_j \frac{\partial \mathbf{u}^{n-j}}{\partial t} $$
which for $k=1$ and $r=0$ looks something along these lines:
$$ \mathbf{u}^{n+1} = \alpha_0 \mathbf{u}^{n} + \alpha_1 \mathbf{u}^{n-1} + \beta_0 \frac{\partial \mathbf{u}^{n}}{\partial t} $$
which takes the current ($n$) and past ($n-1$) information to predict states in the next (future, $n+1$) iterations.

Notice that $\mathbf{F}(\mathbf{u}, t)$ can also involve solving a boundary value problem, similar to our soft mechanics equations.

```python
class TimeStepper(object):
    """ Class for wrapping a timestepper function with other goodies
    """
    def __init__(self, i_x, i_v, i_dt, i_T):
        """ Initialize the timestepper
        """
        # What forcing function are we using?
        self.forcing = None

        # What timestepping algorithm are we using?
        self.timestepper = None

        self.nsteps = int(i_T/i_dt)
        self.dt = i_dt

        if len(i_x) == len(i_v):
            # Same length, corresponding to index, data makes sense
            self.ndim = len(i_x)
            self.x = np.zeros((self.nsteps, self.ndim))
            self.v = np.zeros((self.nsteps, self.ndim))

            # Set initial values
            self.x[0] = np.array(i_x)
            self.v[0] = np.array(i_v)
        else:
            raise RuntimeError('Length of initial velocity and position \
            not matching')

    def set_forcing_function(self, t_func):
        """ Set forcing function to be used
        """
        if type(t_func) is not str:
            try:
                # If not string, try and evaluate the function
                t_func(x[0])
                # If the function works, set this function as forcing
                self.forcing = t_func
            except:
                raise RuntimeError('Provided function cannot be evaluated')
        else:
            if t_func=="harmonic":
                def harmonic(x):
                    return -x
                self.forcing = harmonic

    def error_norm(self):
        """  For testing convergence, defined as a special function """
        if self.forcing.__name__ == 'harmonic':
            time_arr = np.arange(0.0, self.dt*self.nsteps, self.dt)
            analytical_solution = self.x[0, :]*np.exp(-time_arr.reshape(-1,1))
            return np.linalg.norm(analytical_solution - self.x, np.inf)

    def timestep_using(self, timestepper):
        """ Provides access to internal variables x and v
        Applies func over and over again till number of timesteps reached.
        """
        self.timestepper = timestepper.__name__
        for i in range(self.nsteps - 1):
            # Do one cycle
            self.x[i+1], self.v[i+1] = timestepper(self.dt, self.x[i], self.v[i], self.forcing)

    def draw(self, renderer):
        """ Draw the matplotlib canvas with the portrait we want
        """
        if self.timestepper:
            # If there is a timestepper, then there is numerical data
            # Plot them
            renderer.scatter(self.x[0], self.v[0], c='k',marker='o')
            renderer.plot(self.x, self.v, label=self.timestepper)
            x_min, x_max = np.min(self.x), np.max(self.x)
            v_min, v_max = np.min(self.v), np.max(self.v)
            extension = 0.5
            renderer.set_xlim(min(0.0, x_min) - extension, max(0.0, x_max) + extension)
            renderer.set_ylim(min(0.0, x_min) - extension, max(0.0, v_max) + extension)
            renderer.legend()
        else:
            # If there is no timestepper, you are looking for analytical data,
            # if it exists
            # Plot them instead
            if self.forcing.__name__ == 'harmonic':
                true_sol = plt.Circle((0, 0), 1.0, fill=None, edgecolor='k', linestyle='--', lw=4)
                renderer.set_xlim(-1.5, 1.5)
                renderer.set_ylim(-1.5, 1.5)
                renderer.add_artist(true_sol)
            # raise RuntimeError('No information found. Did you forget to run your timestepper?')
        renderer.set_xlabel(r'$x$')
        renderer.set_ylabel(r'$v$')
        renderer.set_title(r'${}$'.format(self.forcing.__name__))
        renderer.set_aspect('equal')

    def draw_sol(self, renderers):
        """ Draw the matplotlib canvas with the solution we want
        """
        time_arr = np.arange(0.0, self.dt*self.nsteps, self.dt)
        if self.timestepper:
            # If there is a timestepper, then there is numerical data
            # Plot them
            renderers[0].plot(time_arr/2.0/np.pi, self.x[:, 0], '-o', label='position')
            renderers[1].plot(time_arr/2.0/np.pi, self.v[:, 0], '-o', label='velocity')
            renderers[2].plot(time_arr/2.0/np.pi, self.x[:, 0]**2 + self.v[:, 0]**2, '-o', label='energy')
            extension = 0.5
            # Plot almost 5 cycles
            renderers[0].set_xlim(-0.05, 5.05)
            renderers[0].set_ylim(-1.05, 1.05)
            renderers[1].set_ylim(-1.05, 1.05)
            renderers[2].set_ylim(0.0, 10.0)
            # renderer_one.legend()
        else:
            # If there is no timestepper, you are looking for analytical data,
            # if it exists
            # Plot them instead
            if self.forcing.__name__ == 'harmonic':
                analytical_pos = np.cos(time_arr.reshape(-1,))
                analytical_vel = -np.sin(time_arr.reshape(-1,))
                renderers[0].plot(time_arr/2.0/np.pi, analytical_pos, 'k--', label=self.timestepper)
                renderers[1].plot(time_arr/2.0/np.pi, analytical_vel, 'k--', label=self.timestepper)
                renderers[2].plot(time_arr/2.0/np.pi, analytical_pos**2 + analytical_vel**2, 'k--', label='energy')
                renderers[0].set_xlim(-0.05, 5.05)
                renderers[0].set_ylim(-1.05, 1.05)
                renderers[1].set_ylim(-1.05, 1.05)
                renderers[2].set_ylim(0.0, 5.0)
            # raise RuntimeError('No information found. Did you forget to run your timestepper?')
        renderers[0].set_ylabel(r'$x(t)$')
        renderers[1].set_ylabel(r'$v(t)$')
        renderers[2].set_ylabel(r'$E(t)$')
        renderers[2].set_xlabel(r'$t/T$')
        renderers[0].set_title(r'Analytical solution')

    def animate(self, fig, renderer, color):
        """ Access to the animate class from matplotlib
        """
        self.timestepper = None

        # animation function. This is called sequentially
        def animate_in(i):
            renderer.clear()
            self.draw(renderer)
            for j in range(i + 1):
                renderer.plot([self.x[j], self.x[j+1]], [self.v[j], self.v[j+1]], marker='o', c=color, alpha=0.5**((i-j)/20.))

        # call the animator. blit=True means only re-draw the parts that have changed.
        anim = animation.FuncAnimation(fig, animate_in, frames=100, interval=5)

        return anim
```

## Time-stepping routines

A variety of numerical algorithms for solving such initial value problems exist, and we are going to look at three main ones : (a) Euler method (or Euler forward) (b) Runge-Kutta-4/RK4 (multi-stage methods) and (c) Position Verlet (symplectic, area preserving) integrators, although others will be discussed on the way. We will attempt to compare these methods in terms of their ease (in understanding/implementation), order of accuracy (in comparison to the time step $dt$), function evaluations for each step $dt$ and some *special* properties.

## Order of accuracy of time-steppers
What's **order** of accuracy? Order of accuracy quantifies the rate of convergence of a numerical approximation of a differential equation to the exact solution.
The numerical solution $\mathbf{u}$ is said to be $n^{\text{th}}$-order accurate if the error, $e(dt):=\lVert\tilde{\mathbf{u}}-\mathbf{u} \rVert$ is proportional to the step-size $ dt $, to the $n^{\text{th}}$ power. That is

$$ e(dt)=\lVert\tilde{\mathbf{u}}-\mathbf{u} \rVert\leq C(dt)^{n} $$

Details of this are given in the slides. Here, we focus on the implementation of these time-steppers for a simple ODE and figure out the order of convergence. The model problem that we deal with is

$$ \frac{dy}{dt} = -y \quad,\quad  y(0) = 1$$
which as we know has the analytical solution $ \tilde{y}(t) = e^{-t} $, so error can be calculated.

### Euler's method
$$ x^{n+1} = x^{n} + f(x^{n})dt $$

```python
# Question
def euler_fwd_ooa(dt, x, v, force_rule):
    """Does one iteration/timestep using Forward Euler scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    # Fill in

    x_n = x
    v_n = v
    return x_n, v_n
```

```python
# Answer
def euler_fwd_ooa(dt, x, v, force_rule):
    """Does one iteration/timestep using Forward Euler scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    # Fill in
    x_n = x + dt * force_rule(x)
    v_n = v
    return x_n, v_n
```

### Euler's method (backward, implicit)
$$ x^{n+1} = x^{n} + \underbrace{f(x^{n+1})dt}_{\text{Evaluated at next timestep!}} $$

```python
# Answer
def euler_bwd_ooa(dt, x, v, force_rule):
    """Does one iteration/timestep using Backward Euler scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    # Fill in
    if force_rule.__name__ == "harmonic":
        x_n = x /(1 + dt)
    else:
        raise NotImplementedError("Cannot do implicit timemarching")
    v_n = v
    return x_n, v_n
```

### Midpoint method
$$ \begin{align}
x^{*}&= x^{n} + f({x}^{n}) \cdot \frac{dt}{2} \\ x^{n+1} &= x + f({x}^{*}) \cdot dt \\
\end{align}
$$

```python
# Question
def midpoint_method_ooa(dt, x , v, force_rule):
    """Does one iteration/timestep using the midpoint scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """

    # Fill in

    x_n = x
    v_n = v
    return x_n, v_n
```

```python
# Answer
def midpoint_method_ooa(dt, x , v, force_rule):
    """Does one iteration/timestep using the midpoint scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """

    # Fill in
    temp_x = x + 0.5*dt*force_rule(x)
    x_n = x + dt * force_rule(temp_x)
    v_n = v
    return x_n, v_n
```

### Runge Kutta-4
$$ \begin{align}
    {k}_1 &= {f}({x}^{n}) \cdot dt \\
	{k}_2 &= {f}({x}^{n} + 0.5 \cdot {k}_1)\cdot dt \\
	{k}_3 &= {f}({x}^{n} + 0.5 \cdot {k}_2)\cdot dt \\
	{k}_4 &= {f}({x}^{n} + {k}_3)\cdot dt \\
	{x}^{n+1} &= {x}^{n} + \frac{{k}_1+2{k}_2+2{k}_3+{k}_4}{6}
\end{align} $$

```python
# Question
def rk4_ooa(dt, x, v, force_rule):
    """Does one iteration/timestep using the RK4 scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    # Fill in

    x_n = x
    v_n = v
    return x_n, v_n
```

```python
# Answer
def rk4_ooa(dt, x, v, force_rule):
    """Does one iteration/timestep using the RK4 scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    # Fill in
    # Stage 1
    k_1 = dt*force_rule(x)

    # Stage 2
    k_2 = dt * force_rule(x + 0.5*k_1)

    # Stage 3
    k_3 = dt * force_rule(x + 0.5*k_2)

    # Stage 4
    k_4 = dt * force_rule(x + k_3)

    x_n = x + (1./6.)*(k_1 + 2.*k_2 + 2.* k_3 + k_4)

    v_n = v
    return x_n, v_n
```

Given your implementation of these schemes, let's implement them on the model problem and see their order of accuracy.

```python
# Initial conditions
i_x = [1.0] # Initial position
i_v = [0.0] # Initial velocity (not required for this problem)
f_T = 10.0 # Final time

# All functions that you coded up
func_list = [euler_fwd_ooa, euler_bwd_ooa, midpoint_method_ooa, rk4_ooa]

# Time steps and associated errors from 2^0 to 2^(-10)
dt_steps = np.arange(11, dtype=np.int16)
errors_list = [[None for i in dt_steps] for func in func_list]

# Run simulations and collect errors
for i_func, func in enumerate(func_list):
    for i_step in dt_steps:
        dt = (2.)**(-i_step)
        b = TimeStepper(i_x, i_v, dt, f_T)
        b.set_forcing_function('harmonic')
        b.timestep_using(func)
        errors_list[i_func][i_step] = b.error_norm()
```

```python
# Draw error plots in a log-log plot
fig, ax = plt.subplots(1,1, figsize=(10, 10))

# x axis is time, y axis is error
for i_func, func in enumerate(func_list):
    ax.plot((2.)**(-dt_steps), errors_list[i_func], 'o-', label=func.__name__)

# Draw helpful slope lines to compare
slopes_list = [None for func in func_list]
slopes_list[0] = 0.1 * (2.)**(-dt_steps)
slopes_list[1] = 0.1 * (2.)**(-dt_steps)
slopes_list[2] = 0.05 * (2.)**(-2*dt_steps)
slopes_list[3] = 0.01 * (2.)**(-4*dt_steps)

for slope_lines in slopes_list:
    ax.plot((2.)**(-dt_steps), slope_lines, 'k--')

# Make it readable
ax.set_xlabel(r'$dt$')
ax.set_ylabel(r'$e(dt)$')
ax.set_title('Order of accuracy')
ax.set_yscale('log')
ax.set_xscale('log')
ax.legend()

# fig.savefig('ooa.pdf')

# Save data if you need to plot it in another application
SAVE_FLAG = True

if SAVE_FLAG:
    import os
    DATA_PATH = os.path.join(os.getcwd(),'data')
    if not os.path.isdir(DATA_PATH):
        os.makedirs(DATA_PATH)

    for i_func, func in enumerate(func_list):
        data_arr = np.vstack(((2.)**(-dt_steps), np.array(errors_list[i_func])))
        np.savetxt(os.path.join(DATA_PATH, func.__name__ + '.txt'), data_arr.T, delimiter='\t')
        data_arr = np.vstack(((2.)**(-dt_steps), (slopes_list[i_func])))
        np.savetxt(os.path.join(DATA_PATH, func.__name__ + '_slopes.txt'), data_arr.T, delimiter='\t')
        # print(data_arr.shape)
        # ax.plot((2.)**(-dt_steps), errors_list[i_func], 'o-', label=func.__name__)
```

Some questions to ponder about
- How do you interpret this diagram?
- Are these schemes robust to coding errors (i.e. replace `x` by `x*` and see what happens)
- Are these bound to exhibit the same behavior, irrespective of $f$?

## Symmetry/Symplectic/Energy preserving characteristics
### Harmonic oscillator
Let's consider the equations governing the dynamics of the harmonic oscillator next. What are harmonic oscillators?

Any undamped linear spring-mass system! (*This statement is not completely true, but for this class it is*).

(Credits: User `kma`, https://tex.stackexchange.com/a/58448)

This system is modeled by $\ddot{x} + x = 0$ (parameters are normalized). We first decompose this second order ODE to a system of two first order ODEs (so that we can use the same first order ODE machinery seen before) by introducing the transformation $ y = \dot{x} $ (If $x$ is the position, $y = \dot{x}$ indicates the velocity. Doing so gives rise to the following linear system, whose dynamics in time we need to uncover:
$$ \begin{pmatrix} \dot{x} \\ \dot{y} \end{pmatrix} = \begin{bmatrix} 0 & 1\\-1 & 0 \end{bmatrix} \begin{pmatrix} {x} \\ {y} \end{pmatrix} $$
wherein we consider the initial conditions to be $ x(0) = 1, y(0) = \dot{x}(0) = 0 $. The analytical solution of this system of ODEs is $ x(t) = \cos(t), y(t) = -\sin(t) $, as seen in class. Notice that if we draw the phase plane of position (in x-axis) and velocity (in y-axis), then
$$ x^2(t) + y^2(t) \equiv 1$$

```python
i_x = [1.0]
i_v = [0.0]

# # First set, fine
dt = 1.0
f_T = 62.8

# Second set, very coarse
# dt = 5.0
# f_T = 628.0

fig2, ax2_list = plt.subplots(3,1, figsize=(10, 10), sharex=True)
fig, ax = plt.subplots(1,1, figsize=(10, 10))
a = TimeStepper(i_x, i_v, dt, f_T)
a.set_forcing_function('harmonic')
a.draw(ax)
a.draw_sol(ax2_list)
# fig.savefig('true_solution.pdf')
```

### Euler's forward method
For a system of ODEs

```python
# Question
def euler_fwd(dt, x, v, force_rule):
    """Does one iteration/timestep using the Euler forward scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    x_n = x
    v_n = v
    return x_n, v_n

a.timestep_using(euler_fwd)
```

```python
# Answer
def euler_fwd(dt, x, v, force_rule):
    """Does one iteration/timestep using the Euler forward scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    x_n = x + dt * v
    v_n = v + dt * force_rule(x)
    return x_n, v_n

a.timestep_using(euler_fwd)
```

```python
a.draw_sol(ax2_list)
fig2
```

```python
a.draw(ax)
fig
# fig.savefig('euler_fwd_1.0.pdf')
```

### Position verlet scheme for second order ODE
$$ \begin{align}
	\mathbf{x}^* &= \mathbf{x}^n + 0.5\cdot dt \cdot \mathbf{y}^n \\
	\mathbf{y}^{n+1} &= \mathbf{y}^n + dt \cdot \mathbf{f}\left( \mathbf{x}^*\right) \\
	\mathbf{x}^{n+1} &= \mathbf{x}^* + 0.5\cdot dt \cdot \mathbf{y}^{n+1}
\end{align} $$
We only have **one** functional evaluation, while obtaining **second**-order accuracy in position and velocity.

```python
# Question
def position_verlet(dt, x, v, force_rule):
    """Does one iteration/timestep using the Position verlet scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    v_n = v
    x_n = x
    return x_n, v_n

a.timestep_using(position_verlet)
```

```python
# Answer
def position_verlet(dt, x, v, force_rule):
    """Does one iteration/timestep using the Position verlet scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    temp_x = x + 0.5*dt*v
    v_n = v + dt * force_rule(temp_x)
    x_n = temp_x + 0.5 * dt * v_n
    return x_n, v_n

a.timestep_using(position_verlet)
```

```python
a.draw_sol(ax2_list)
fig2
```

```python
a.draw(ax)
fig
```

### Velocity verlet scheme for second order ODE
$$ \begin{align}
	\mathbf{y}^* &= \mathbf{y}^n + 0.5\cdot dt \cdot \mathbf{f}\left( \mathbf{x}^n\right) \\
	\mathbf{x}^{n+1} &= \mathbf{x}^{n} + dt \cdot \mathbf{y}^{*} \\
\mathbf{y}^{n+1} &= \mathbf{y}^* + 0.5\cdot dt \cdot \mathbf{f}\left( \mathbf{x}^{n+1}\right) \\
\end{align} $$
Note that we now have two functional evaluations, while retaining second-order accuracy in position and velocity.

```python
# Question
def velocity_verlet(dt, x, v, force_rule):
    """Does one iteration/timestep using the Velocity verlet scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    x_n = x
    v_n = v
    return x_n, v_n

a.timestep_using(velocity_verlet)
```

```python
# Answer
def velocity_verlet(dt, x, v, force_rule):
    """Does one iteration/timestep using the Velocity verlet scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    temp_v = v + 0.5*dt*force_rule(x)
    x_n = x + dt * temp_v
    v_n = temp_v + 0.5* dt * force_rule(x_n)
    return x_n, v_n

a.timestep_using(velocity_verlet)
```

```python
a.draw_sol(ax2_list)
fig2
```

```python
a.draw(ax)
fig
```

### Euler-Cromer scheme
The second variant of semi-implicit Euler/Euler-Cromer scheme is
$$ \begin{align}
\mathbf{x}^{n+1} &= \mathbf{x}^{n} + dt \cdot \mathbf{y}^{n+1} \\
\mathbf{y}^{n+1} &= \mathbf{y}^{n} + dt \cdot \mathbf{f}\left( \mathbf{x}^{n+1}\right) \\
\end{align} $$

```python
# Question
def euler_cromer(dt, x, v, force_rule):
    """Does one iteration/timestep using the Euler Cromer scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    x_n = x
    v_n = v
    return x_n, v_n

# a.timestep_using(euler_cromer)
```

```python
# Answer
def euler_cromer(dt, x, v, force_rule):
    """Does one iteration/timestep using the Euler Cromer scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    x_n = x + dt * v
    v_n = v + dt * force_rule(x_n)
    return x_n, v_n

# a.timestep_using(euler_cromer)
```

### Runge Kutta-4
For a system of ODEs. Repeated again for convenience
$$ \begin{align}
	\mathbf{k}_1 &= \mathbf{f}(\mathbf{x}^{n}) \cdot dt \\
	\mathbf{k}_2 &= \mathbf{f}(\mathbf{x}^{n} + 0.5 \cdot \mathbf{k}_1)\cdot dt \\
	\mathbf{k}_3 &= \mathbf{f}(\mathbf{x}^{n} + 0.5 \cdot \mathbf{k}_2)\cdot dt \\
	\mathbf{k}_4 &= \mathbf{f}(\mathbf{x}^{n} + \mathbf{k}_3)\cdot dt \\
	\mathbf{x}^{n+1} &= \mathbf{x}^{n} + \frac{\mathbf{k}_1+2\mathbf{k}_2+2\mathbf{k}_3+\mathbf{k}_4}{6}
\end{align} $$

```python
# Question
def runge_kutta4(dt, x, v, force_rule):
    """Does one iteration/timestep using the RK4 scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """
    x_n, v_n = x, v
    return x_n, v_n

a.timestep_using(runge_kutta4)
```

```python
# Answer
def runge_kutta4(dt, x, v, force_rule):
    """Does one iteration/timestep using the RK4 scheme

    Parameters
    ----------
    dt : float
        Simulation timestep in seconds
    x : float/array-like
        Quantity of interest / position of COM
    v : float/array-like
        Quantity of interest / velocity of COM
    force_rule : ufunc
        A function, f, that takes one argument and
        returns the instantaneous forcing

    Returns
    -------
    x_n : float/array-like
        The quantity of interest at the Next time step
    v_n : float/array-like
        The quantity of interest at the Next time step
    """

    def vector_func(y):
        return np.array([y[1], force_rule(y[0])])

    # Base
    u = np.array([x,v])

    # Stage 1
    k_1 = dt*vector_func(u)

    # Stage 2
    k_2 = dt * vector_func(u + 0.5*k_1)

    # Stage 3
    k_3 = dt * vector_func(u + 0.5*k_2)

    # Stage 4
    k_4 = dt * vector_func(u + k_3)

    u_n = u + (1./6.)*(k_1 + 2.*k_2 + 2.* k_3 + k_4)

    return u_n[0], u_n[1]

a.timestep_using(runge_kutta4)
```

```python
a.draw_sol(ax2_list)
fig2
```

```python
a.draw(ax)
fig
# fig.savefig('rk4_1.0.pdf')
```

Some questions to ponder about
- What do you observe as you increase \( dt \) in all cases?
- Are symplectic schemes perfect in its descrption of physics?

# Video of the maps, please ignore

```python
fig3, ax3 = plt.subplots(1,1, figsize=(10, 10))
i_x = [0.0]
i_v = [1.0]
dt = 1.0
f_T = 700.0
b = TimeStepper(i_x, i_v, dt, f_T)
b.set_forcing_function('harmonic')
b.draw(ax2)
b.timestep_using(position_verlet)
ax3.set_xlim([-1.5, 1.5])
ax3.set_ylim([-1.5, 1.5])
#b.draw(ax2)
# fig2
```

```python
cmap = sns.color_palette()
verlet_color = cmap[0]
rk_color = cmap[2]
anim = b.animate(fig3, ax3, verlet_color)
```

```python
HTML(anim.to_jshtml())
```

```python
# anim.save('verlet.mp4', fps=30,
#           extra_args=['-vcodec', 'h264',
#                       '-pix_fmt', 'yuv420p'])
```


```python
import jax.numpy as jnp
from jax import grad, jit, vmap
from jax import jacfwd, jacrev
from jax import random
import numpy as np
import sympy as sym
import matplotlib.pyplot as plt

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.animation as animation
from celluloid import Camera
import time

class Visualization:
    def __init__(self, X_opt, cost, lb, ub):
        self.X_opt = X_opt
        self.cost = cost
        self.lb = lb
        self.ub = ub

        self.f = plt.figure(figsize=(12, 8))
        plt.rc('axes', linewidth=2)

        x = np.arange(lb, ub, 0.1)
        self.X, self.Y = np.meshgrid(x, x)
        M = {0: self.X, 1: self.Y}
        self.F = self.cost(M)

    def plot_optimization(self):
        plt.plot([self.X_opt[i][0] for i in range(len(self.X_opt))],
                 [self.X_opt[i][1] for i in range(len(self.X_opt))], '-*',
                 alpha=0.5, color='white', markersize=15, lw=2.0)
        plt.contourf(self.X, self.Y, self.F)
        plt.xlabel('$x_1$', size=25)
        plt.ylabel('$x_2$', size=25)
        plt.show()
        # f.savefig("steepest_descent_armijo.pdf", bbox_inches='tight')

    def animate_optimization_2D(self):

        camera = Camera(plt.figure(figsize=(10, 8)))

        # plot contour of the function
        # plt.contour(X, Y, F)
        # policy
        x1 = [X_opt[i][0] for i in range(len(X_opt))]
        x2 = [X_opt[i][1] for i in range(len(X_opt))]

        for i in range(len(x1)):
            plt.contourf(X, Y, F)
            plt.plot(x1[0], x2[0], '*', color='white', markersize=20, lw=2.0, alpha=0.5)
            x1data, x2data = x1[:i+1], x2[:i+1]
            plt.plot(x1data, x2data, '-', color='white', markersize=10, lw=2.0)
            plt.plot(x1data[-1], x2data[-1], 'o', color='white', markersize=15, lw=2.0, alpha=0.5)
            plt.text(1.0, 1.75, r'iteration: {}'.format(len(x1data)), fontsize=18)
            plt.xlabel('$x_1$', size=25)
            plt.ylabel('$x_2$', size=25)
            camera.snap()

        anim = camera.animate(blit=True)
        anim.save('optimization_animation.gif', writer='imagemagick')
        plt.show()

    def animate_optimization_3D(self):
        fig = plt.figure()
        ax = fig.add_subplot(111, projection='3d')

        camera = Camera(plt.figure(figsize=(10, 8)))

        x1 = [X_opt[i][0] for i in range(len(X_opt))]
        x2 = [X_opt[i][1] for i in range(len(X_opt))]

        for i in range(len(x1)):
            plt.contour(X, Y, F)
            plt.plot(x1[0], x2[0], '*', color='red', markersize=20, lw=2.0, alpha=0.5)
            x1data, x2data = x1[:i+1], x2[:i+1]
            plt.plot(x1data, x2data, '-', color='red', markersize=10, lw=2.0)
            plt.plot(x1data[-1], x2data[-1], 'o', color='red', markersize=10, lw=2.0)
            plt.xlabel('$x_1$', size=25)
            plt.ylabel('$x_2$', size=25)
            camera.snap()

        anim = camera.animate(blit=True)
        anim.save('optimization_animation.gif', writer='imagemagick')
        plt.show()

class UnconstrainedOptimization:
    def __init__(self, cost, direction_type, stepsize_type, x0,
                 termination_params, algo_params=None):
        self.cost = cost
        self.x0 = jnp.array(x0)
        self.direction_type = direction_type
        self.stepsize_type = stepsize_type
        self.algo_params = algo_params
        self.termination_params = termination_params

    def armijo_condition(self, alpha, sigma, x_current, x_next, direction):
        LHS = self.cost(x_current) - self.cost(x_next)
        RHS = - sigma * alpha * jnp.matmul(grad(self.cost)(x_current), direction)
        condition = True if (LHS >= RHS) else False
        return condition

    def get_step_size(self, x_current, direction):

        if self.stepsize_type == 'minimization_rule':
            # create symbolic step size: alpha
            alpha = sym.Symbol('alpha')
            # convert jax device array to numpy list
            x_current = x_current.tolist()
            direction = direction.tolist()
            sym_vec = []
            for d in range(len(direction)):
                mm = x_current[d] + alpha * direction[d]
                sym_vec.append(mm)
            sym_cost = self.cost(sym_vec)
            sym_derivative = sym.diff(sym_cost, alpha)
            alpha_sym = sym.solveset(sym_derivative, alpha)
            alpha_sym = list(alpha_sym)

            # only keep the real number by checking data type
            i = 0
            while True:
                condition = type(alpha_sym[i] ** 2) is sym.numbers.Float
                if condition is True:
                    val = alpha_sym[i]
                    break
                else:
                    i += 1

            alpha = float(val)

        elif self.stepsize_type == 'constant':
            alpha = self.algo_params

        elif self.stepsize_type == 'armijo_rule':
            beta, s, sigma = self.algo_params
            m = 0
            condition = False
            while True:
                alpha = pow(beta, m) * s
                x_next = x_current + alpha * direction
                condition = self.armijo_condition(alpha, sigma, x_current, x_next, direction)

                if condition is True:
                    break
                m += 1
        else:
            raise Exception(
                'Unknown step size type, please input one of the following: minimization_tule, constant, armijo rule')

        return alpha

    def get_gradient_direction(self, x_val):
        # get derivative of the cost function
        derivative = grad(self.cost)(x_val)

        if self.direction_type == 'steepest_descent':
            # D in case of steepest descent is identity matrix
            D = jnp.identity(len(x_val))
            direction = - jnp.matmul(D, derivative)

        elif self.direction_type == 'newton_method':
            hessian = jacfwd(jacrev(self.cost))(x_val)
            D = jnp.linalg.inv(hessian)
            direction = - jnp.matmul(D, derivative)

        else:
            raise Exception(
                'Unknown gradient based method, please input one of the following: '
                'steepest_descent, newton_method')

        return direction

    def update(self, x_current, X_opt, error_val, error_cache):

        # update states
        X_opt.append(np.asarray(x_current, dtype=float))
        direction = self.get_gradient_direction(x_current)
        step_size = self.get_step_size(x_current, direction)
        x_next = x_current + step_size * direction
        error_val = x_current - x_next
        error_norm = jnp.linalg.norm(error_val)
        error_cache.append(error_norm)

        return x_next, X_opt, error_norm, error_cache

    def run_algorithm(self):
        """
        Runs the algorithm provided the user with starting and termination conditions,
        If the termination condition is "convergence" then the algorithm runs until convergence
        is achieved upto the desired value, the other termination option is to run the algorithm for
        a fixed number of timesteps.
        :return: X_opt = a matrix with co-ordinates of all the states obtained from running the algorithm
        """

        # get termination conditions from algo_params
        termination_type, termination_condition = self.termination_params

        x_current = self.x0
        X_opt = []
        error_norm = np.inf
        error_cache = []
        fig = plt.figure()

        if termination_type == "convergence":
            while error_norm > termination_condition:
                x_next, X_opt, error_norm, error_cache = self.update(
                    x_current, X_opt, error_norm, error_cache)
                # update state vector
                x_current = x_next

        elif termination_type == "fixed_steps":
            for t in range(termination_condition):
                x_next, X_opt, error_norm, error_cache = self.update(
                    x_current, X_opt, error_norm, error_cache)
                # update state vector
                x_current = x_next

        return X_opt, error_cache
```

```python

cost = lambda x: 3*pow(x[0], 2) + pow(x[1], 4)

x0 = [0.5, 1.5]
algo_params = 0.10 #[0.5, 1, 0.1]
opt = UnconstrainedOptimization(cost, 'steepest_descent', 'minimization_rule', x0,
                                ['fixed_steps', 50], algo_params)
X_opt, error_cache = opt.run_algorithm()
viz = Visualization(X_opt, cost, -1.5, +1.5)
viz.animate_optimization_2D()

```

```python
cost = lambda x: 3*pow(x[0], 2) + pow(x[1], 4)

x = np.arange(-2.0, +2.0, 0.1)
X, Y = np.meshgrid(x, x)
M = {0: X, 1: Y}
F = cost(M)

fig = plt.figure(figsize=(10, 8))
ax = plt.axes(projection='3d')
ax.view_init(45, 30)
camera = Camera(fig)

# plot contour of the function
# plt.contour(X, Y, F)
# policy
x1 = [X_opt[i][0] for i in range(len(X_opt))]
x2 = [X_opt[i][1] for i in range(len(X_opt))]
f_opt = [cost(X_opt[i]) for i in range(len(X_opt))]

ax.scatter3D(x1data[0], x2data[0], f_opt[0], 'o', color='black', lw=2.0)
for i in range(len(x1)):
    x1data, x2data = x1[:i+1], x2[:i+1]
    ax.plot3D(x1data, x2data, f_opt[:i+1], '-', color='black', lw=2.0)
    ax.scatter3D(x1data[-1], x2data[-1], f_opt[-1], 'o', color='black', lw=2.0)
    ax.plot_surface(X, Y, F, rstride=1, cstride=1,
            cmap='viridis', edgecolor='none')

    camera.snap()

anim = camera.animate(blit=True)
anim.save('optimization_animation.gif', writer='imagemagick')
plt.show()
```

```python

```

```python

```

```python

```

```python

```


# COMP 135 day07: Probabilistic Linear Regression

## Outline

* **Part 1: Showing samples from the posterior**
* * Learn to make plots of posterior distributions over the weights for simple regression problems

## Takeaways

* Hyperparameters like $\alpha$ (prior precision) and $\beta$ (likelihood precision) matter
* * We'll talk more about dealing with these next class

```python
import numpy as np
import pandas as pd
import scipy.stats
```

```python
np.set_printoptions(precision=3, suppress=False)
```

```python
pd.options.display.float_format = '{:,.3g}'.format  # show 3 digits of precision
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("notebook", font_scale=1.25)
```

```python

```

# Part 1: The Probabilistic view of linear regression

### Task: Linear regression

Given $N$ observations of *paired* feature-outcome observations: $\{ x_n, t_n \}$. Each $x_n$ is a scalar, and each $t_n$ is a scalar.

We're also given a feature transform function $\phi$ which maps each $x_n$ to a vector in $M$-dimensional space. This function is known in advance. For now, we'll assume that the "feature transform" $\phi(x_n)$ just simply passes along the features $x_n$, while adding an additional offset or "intercept" feature that is always 1. This is a *simplifying* assumption for today.

We want to make good predictions of new outcomes $t_*$ given new features $x_*$.

```python
def calc_features(x_N1, M=2):
    ''' Transform raw features into complete features useful for prediction

    Could do any non-linear transformations thought relevant for the problem.
    Here we'll just do an identity transform with an extra intercept feature.

    Args
    ----
    x_N1 : 2D array, shape (N, 1) = (n_examples,)

    Returns
    -------
    phi_NM : 2D array, shape (N, M) = (n_examples, n_transformed_features)
        First column will contain all ones (a bias or intercept feature)
        Second column will just include the raw features
    '''
    assert x_N1.ndim == 2
    assert x_N1.shape[1] == 1
    N = x_N1.shape[0]
    phi_NM = np.zeros((N, M))
    phi_NM[:,0] = 1
    phi_NM[:,1] = x_N1[:,0]
    return phi_NM
```

## Exercise 1a: Obtain transformed features $\phi(x_n)$ when $x_n = [2.0]$

```python
# TODO call calc_features() for the vector x_n = [2.0]
```

## Define the Likelihood

Each observation (indexed by $n$) is drawn iid from a Normal as follows:

$$
t_n \sim \mathcal{N}( t_n | w^T \phi(x_n), \beta^{-1} )
$$

The key properties here are:
* The *mean* is a linear function of the transformed features.
* The *precision* (inverse variance) is assumed a known value defined by parameter $\beta > 0$

## Define the  Prior

For now, we'll assume that weights come from a zero mean prior with some covariance determined by a scalar parameter $\alpha$:

$$
w \sim \mathcal{N}( 0, \alpha^{-1} I_M )
$$

A zero mean prior makes sense if we don't know if the slope should be negative or positive.

### Parameter we'll treat as a random variable: $w$

* Weights vector: $w = [w_1, w_2, \ldots w_M]^T$, so $w \in \mathbb{R}^M$

### Parameters we'll treat as fixed: $\alpha$ and $\beta$

* Prior precision $\alpha > 0$

The larger $\alpha$ is, the more confident we are in the weight values before seeing any data.

* Likelihood precision $\beta > 0$

The larger $\beta$ is, the less the observed $t$ values vary from the predicted mean.

## Create a simple toy data for analysis

Just execute the cells below to get the sense of how to generate toy data from this model

We'll manually intervene to set the weight vector to a known value. This makes it easy to tell if our learning is working later on.

```python
N = 2000  # Number of examples we observe
M = 2     # Number of transformed features
```

```python
# Create reproducible pseudorandom number generator
prng = np.random.RandomState(0)
```

Create the weight vector we'll use to generate our dataset. Set an intercept of 1.2 and a slope of -0.75

```python
true_w_M = np.asarray([1.2, -0.75])
```

Create a "true" alpha value which controls the prior precision

```python
true_alpha = 0.01
```

Create a "true" beta which defines the likelihood precision

```python
true_beta = 0.5
```

Create observed features $x$ and do a "feature transform" that just adds a column of all ones (an "intercept" feature)

```python
x_N1 = prng.uniform(low=-7, high=7, size=N)[:,np.newaxis]
phi_NM = calc_features(x_N1)
```

Generate outputs t using our assumed model

```python
t_N = prng.normal( np.dot(phi_NM, true_w_M), 1.0/true_beta)
```

## Visualize the toy dataset

```python
plt.plot(x_N1, t_N, 'k.');

ax_h = plt.gca()
ax_h.set_aspect('equal', 'box');
ax_h.set_xlim([-8, 8]); ax_h.set_xticks([-6, -4, -2, 0, 2, 4, 6]);
ax_h.set_ylim([-8, 8]); ax_h.set_yticks([-6, -4, -2, 0, 2, 4, 6]);

xgrid_G = np.linspace(-8, 8, 100)[:,np.newaxis]
plt.plot(xgrid_G, np.dot(calc_features(xgrid_G), true_w_M), 'c-', linewidth=3);

plt.xlabel('input: x');
plt.ylabel('output: t');
plt.title("Toy Data\n true_slope %.2f \n true intercept %.2f\n true precision beta %.2f" % (
    true_w_M[1], true_w_M[0], true_beta));
```

## Posterior over weights

Recall the formulas for the posterior over the weight vector $w$, simplified for our zero-mean, diagonal-covariance prior on $w$:

\begin{align}
p( w | t) &= \text{MVNormPDF}( w | m, S )
\\
S^{-1} &= \alpha I_M + \beta \Phi^T \Phi
\\
m &= \beta S \Phi^T \mathbf{t}
\end{align}

In other words, the posterior is a Gaussian distribution, where:
* $m$ is the mean vector of shape $(M,)$
* $S$ is the covariance matrix of shape $(M, M)$

The ingredients of these formulas are:
* $\alpha > 0$ : prior precision
* $\beta > 0$ : likelihood precision
* $\Phi$ : N x M matrix of transformed features ("inputs")
* $\mathbf{t}$ : N-dimensional vector of observed outputs
* $I_M $: M x M [identity matrix](https://en.wikipedia.org/wiki/Identity_matrix) (can create with `np.eye(M)`)

## Exercise 1b: Compute the parameters of the posterior

Translate the formulas above into concrete NumPy expressions that compute the mean $m$ and covariance $S$:m

```python
def calc_parameters_of_posterior(phi_NM, t_N, alpha=0.1, beta=0.1):
    ''' Compute parameters of the posterior for weights w

    Args
    ----
    phi_NM : 2D array, shape (N,M)
        Each row is a feature vector for one example
    t_N : 1D array, shape (N,)
        Each row is a output scalar value for one example
    alpha : positive scalar
        Prior precision
    beta : positive scalar
        Likelihood precision

    Returns
    -------
    m_M : posterior mean vector, shape (M,)
    S_MM : posterior covariance matrix, shape (M, M)
    '''
    N, M = phi_NM.shape
    invS_MM = np.eye(M) # TODO determine the inverse of S via formulas above
    S_MM = np.linalg.inv(invS_MM) # Invert to get the value of S (covariance matrix)

    m_M = np.zeros(M) # TODO determine the mean vector via formulas above
    return m_M, S_MM
```

## Exercise 1c: Compute posterior parameters when given all N=2000 observations

How close is the posterior mean to the "true" vector of weights that generated the data?

How "uncertain" is the posterior (look at variance values in the covariance matrix)

```python
# TODO call calc_parameters_of_posterior(...) with the toy dataset we generated above
# Use the "true" alpha/beta values

# TODO inspect the mean and covariance that is returned. Does it make sense?
```

## Exercise 1d: Visualize samples from the posterior as more data is observed

Do you see the posterior "concentrate" its mass on the true value as more data is observed?

```python
# Define the number of observations to look at to make a posterior
N_values = [0, 2, 8, 32, 128, N]
C = len(N_values)

# Define the number of samples from the posterior to visualize
R = 10

fig_h, axgrid = plt.subplots(nrows=1, ncols=C, sharex=True, sharey=True, figsize=(C * 4, 4))

for col_id, Ncur in enumerate(N_values):

    # Create the posterior from the first Ncur observations
    # Hint: use phi_NM[:10] to get the first 10 observations
    m_M, S_MM = np.zeros(M), np.eye(M) # TODO call calc_parameters_of_posterior(...)

    # Activate current subplot and get its handle
    ax_h = axgrid[col_id]

    # Show the observed data
    ax_h.plot(x_N1[:Ncur], t_N[:Ncur], linestyle='', marker='s', color='black')

    for rr in range(R):
        # Create a uniform grid of possible x values, to use in making predictions
        xgrid_G = np.linspace(-8, 8, 100)[:,np.newaxis]

        # Draw one sample vector from the posterior with the parameters defined above
        wsamp_rr_M = m_M # TODO call prng.multivariate_normal with mean m_M and covar S_MM

        # Produce the output values *t* that we get from the current sample of w_M
        # after multiplying that weight vector with our features
        tsamp_rr_G = np.dot(calc_features(xgrid_G), wsamp_rr_M)

        # Show a plot of this sample on current axis
        ax_h.plot(xgrid_G, tsamp_rr_G, '-');

    # Make plots pretty and standardized
    ax_h.set_aspect('equal', 'box');
    ax_h.set_xlim([-8, 8]); ax_h.set_xticks([-6, -4, -2, 0, 2, 4, 6]);
    ax_h.set_ylim([-8, 8]); ax_h.set_yticks([-6, -4, -2, 0, 2, 4, 6]);
    ax_h.set_xlabel('input $x$')
    if col_id == 0:
        ax_h.set_ylabel('output $t$')
    ax_h.set_title('after N=%d' % Ncur)
```

## Exercise 1e: Visualize samples from posterior if we use WRONG values for $\beta$

The true beta value we used to generate the data was $\beta = 0.5$

What if instead we use $\beta = 10$.... what will happen?

Compare and contrast with the above in 1d.

```python
wrong_beta = 10.0

# Define the number of observations to look at to make a posterior
N_values = [0, 2, 8, 32, 128, N]
C = len(N_values)

# Define the number of samples from the posterior to visualize
R = 10

# TODO copy code from above, but use *wrong beta* instead of *true_beta*
```

```python

```

```python

```


# Refinery

## Objective and Prerequisites

This model is an example of a production planning problem. In such problems, decisions must be made regarding which products to produce, and which resources to use to produce those products. These problems are common across a broad range of manufacturing and mining industries. In this example, we will build a model to optimize the output of a refinery.

The company can buy a restricted amount of two different kinds of crude oils, which can be processed to gasoline (motor fuel), jet fuel, or oil. The aim is to create an optimal production plan that maximizes the total profit, while considering production capacity and other restrictions.

More information on this type of model can be found in example # 6 of the fifth edition of Modeling Building in Mathematical Programming by H. P. Williams on pages 258 and 306-310.

This modeling example is at the intermediate level, where we assume that you know Python and are familiar with the Gurobi Python API. In addition, you should have some knowledge about building mathematical optimization models.

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

---
## Problem Description

An oil refinery purchases two types of crude oil (Crude 1 and Crude 2) and refines them through a four-step process (distillation, reforming, cracking, and blending) to produce gas and fuels that are sold.

### Step One: Distillation

The distillation process separates each crude oil into six fractions according to their boiling points. These fractions are: light naphtha, medium naphtha, heavy naphtha, light oil, heavy oil, and residuum. The octane numbers for the light, medium and heavy naphthas are, respectively, 90, 80, and 70.

One barrel of crude separates into following fractions:

|  | Light Naphta | Medium Naphta | Heavy Naphta | Light Oil | Heavy Oil | Residuum |
| --- | --- | --- | --- | --- | --- | --- |
| Crude 1 | 0.1 | 0.2 | 0.2 | 0.12 | 0.2 | 0.13 |
| Crude 2 | 0.15 | 0.25 | 0.18 | 0.08 | 0.19 | 0.12 |

There is a small amount of waste in distillation (5% and 3% for Crude 1 and 2, respectively).

### Step Two: Reforming

After distillation, the resulting naphthas can be blended into different grades of gas, or they can go through a process called reforming. The output of the reforming process is a product known as reformed gasoline with an octane number of 115.

The yields of reformed gasoline from each barrel of the different naphthas are given as follows:

- One barrel of light naphtha yields 0.6 barrels of reformed gasoline.
- One barrel of medium naphtha yields 0.52 barrels of reformed gasoline.
- One barrel of heavy naphtha yields 0.45 barrels of reformed gasoline.

### Step Three: Cracking

Light and heavy oils can be blended into jet fuel or be put through a process known as catalytic cracking. The catalytic cracker produces cracked oil and cracked gasoline.

Cracked gasoline has an octane number of 105 with the following yields:

- One barrel of light oil yields 0.68 barrels of cracked oil and 0.28 barrels of cracked gasoline.
- One barrel of heavy oil yields 0.75 barrels of cracked oil and 0.2 barrels of cracked gasoline.

Cracked oil is used for blending fuel oil and jet fuel; cracked gasoline is used for blending gasoline. Residuum can be used for either producing lube-oil or blending into jet fuel and fuel oil:

- One barrel of residuum yields 0.5 barrels of lube-oil.

### Step Four: Blending

#### Gasoline

There are two kinds of gasoline — regular and premium — made by blending naphtha, reformed gasoline, and cracked gasoline. The only requirement is that regular gasoline must have an octane of at least 84 and premium gasoline must have an octane number of at least 94. It is assumed that octane numbers blend linearly by volume.

#### Jet Fuel

Jet fuel must have a vapor pressure that does not exceed $1.0 \frac{\text{kg}}{\text{cm}^2}$. The vapor pressures for light, heavy, cracked oils and residuum are, respectively, 1.0, 0.6, 1.5 and 0.05 $\frac{\text{kg}}{\text{cm}^2}$. It is assumed that vapor pressures blend linearly by volume.

#### Fuel Oil

To produce fuel oil, you must blend light oil, cracked oil, heavy oil and residuum in the ratio of 10:4:3:1.

The availability and capacity limitations are as follows:

- The daily availability of crude 1 is 20 000 barrels.
- The daily availability of crude 2 is 30 000 barrels.
- At most 45 000 barrels of crude can be distilled per day.
- At most 10 000 barrels of naphtha can be reformed per day.
- At most 8000 barrels of oil can be cracked per day.
- The daily production of lube oil must be between 500 and 1000 barrels.
- Premium gasoline production must be at least 40% of regular gasoline production.

Each final product has the following profit contribution per barrel (pennies per barrel):

|  | Profit Contribution |
| --- | --- |
| Premium Gasoline | 700 |
| Regular Gasoline | 600 |
| Jet Fuel | 400 |
| Fuel Oil | 350 |
| Lube Oil | 150 |

The key question is: How should the operations of the refinery be planned in order to maximize total profit?

---
## Model Formulation

### Sets and Indices

$i \in \text{Crudes}=\{1,2\}$: Set of crude oils.

### Parameters

$\text{buy_limit}_i \in \mathbb{N}$: Maximum number of barrels of crude $i$ to buy.

$\text{distill_cap} \in \mathbb{N}$: Maximum number of barrels of crude oil to distill.

$\text{reform_cap} \in \mathbb{N}$: Maximum number of barrels of naphtha to reform.

$\text{crack_cap} \in \mathbb{N}$ Maximum number of barrels of oil to crack.

$\text{LBO_min}, \text{LBO_max} \in \mathbb{N}$ Minimum and Maximum number of barrels of lube oil to produce.

### Decision Variables

$\text{CR}_i \in [0,\text{buy_limit}_i] \subset \mathbb{R}^+$: Number of barrels of crude $i$ to buy.

$\text{LN} \in \mathbb{R}^+$: Number of barrels of light naphtha to distill.

$\text{MN} \in \mathbb{R}^+$: Number of barrels of medium naphtha to distill.

$\text{HN} \in \mathbb{R}^+$: Number of barrels of heavy naphtha to distill.

$\text{LO} \in \mathbb{R}^+$: Number of barrels of light oil to distill.

$\text{HO} \in \mathbb{R}^+$: Number of barrels of heavy oil to distill.

$\text{R} \in \mathbb{R}^+$: Number of barrels of residuum to distill.

$\text{LNRG} \in \mathbb{R}^+$: Number of barrels of light naphtha used to produce reformed gasoline.

$\text{MNRG} \in \mathbb{R}^+$: Number of barrels of medium naphtha used to produce reformed gasoline.

$\text{HNRG} \in \mathbb{R}^+$: Number of barrels of heavy naphtha used to produce reformed gasoline.

$\text{RG} \in \mathbb{R}^+$: Number of barrels of reformed gasoline to produce.

$\text{LOCGO} \in \mathbb{R}^+$: Number of barrels of light oil used to produce cracked gasoline and cracked oil.

$\text{HOCGO} \in \mathbb{R}^+$: Number of barrels of heavy oil used to produce cracked gasoline and cracked oil.

$\text{CG} \in \mathbb{R}^+$: Number of barrels of cracked gasoline to produce.

$\text{CO} \in \mathbb{R}^+$: Number of barrels of cracked oil to produce.

$\text{LNPMF} \in \mathbb{R}^+$: Number of barrels of light naphtha used to produce premium motor fuel.

$\text{LNRMF} \in \mathbb{R}^+$: Number of barrels of light naphtha used to produce regular motor fuel.

$\text{MNPMF} \in \mathbb{R}^+$: Number of barrels of medium naphtha used to produce premium motor fuel.

$\text{MNRMF} \in \mathbb{R}^+$: Number of barrels of medium naphtha used to produce regular motor fuel.

$\text{HNPMF} \in \mathbb{R}^+$: Number of barrels of heavy naphtha used to produce premium motor fuel.

$\text{HNRMF} \in \mathbb{R}^+$: Number of barrels of heavy naphtha used to produce regular motor fuel.

$\text{RGPMF} \in \mathbb{R}^+$: Number of barrels of reformed gasoline used to produce premium motor fuel.

$\text{RGRMF} \in \mathbb{R}^+$: Number of barrels of reformed gasoline used to produce regular motor fuel.

$\text{CGPMF} \in \mathbb{R}^+$: Number of barrels of cracked gasoline used to produce premium motor fuel.

$\text{CGRMF} \in \mathbb{R}^+$: Number of barrels of cracked gasoline used to produce regular motor fuel.

$\text{LOJF} \in \mathbb{R}^+$: Number of barrels of light oil used to produce jet fuel.

$\text{HOJF} \in \mathbb{R}^+$: Number of barrels of heavy oil used to produce jet fuel.

$\text{RJF} \in \mathbb{R}^+$: Number of barrels of residuum used to produce jet fuel.

$\text{COJF} \in \mathbb{R}^+$: Number of barrels of cracked oil used to produce jet fuel.

$\text{RLBO} \in \mathbb{R}^+$: Number of barrels of residuum used to produce lube oil.

$\text{PMF} \in \mathbb{R}^+$: Number of barrels of premium motor fuel to produce.

$\text{RMF} \in \mathbb{R}^+$: Number of barrels of regular motor fuel to produce.

$\text{JF} \in \mathbb{R}^+$: Number of barrels of jet fuel to produce.

$\text{FO} \in \mathbb{R}^+$: Number of barrels of fuel oil to produce.

$\text{LBO} \in [\text{LBO_min}, \text{LBO_max}] \subset \mathbb{R}^+$: Number of barrels of lube oil to produce.

### Objective Function

- **Profit:** Maximize the total profit (in hundreds of USD).

\begin{equation}
\text{Max} \quad Z = 7*\text{PMF} + 6*\text{RMF} + 4*\text{JF} + 3.5*\text{FO} + 1.5*\text{LBO}
\tag{0}
\end{equation}

### Constraints

- **Distillation Capacity**: The number of barrels of crude oil to distill cannot exceed the capacity.

\begin{equation}
\sum_{i \in \text{Crudes}}{\text{CR}_i} \leq \text{distill_cap}
\tag{1}
\end{equation}

- **Reforming Capacity**: The number of barrels of naphtha to reform cannot exceed the capacity.

\begin{equation}
\text{LNRG} + \text{MNRG} + \text{HNRG} \leq \text{reform_cap}
\tag{2}
\end{equation}

- **Cracking Capacity**: The number of barrels of oil to crack cannot exceed the capacity.

\begin{equation}
\text{LOCGO} + \text{HOCGO} \leq \text{crack_cap}
\tag{3}
\end{equation}

- **Yield**: The number of barrels produced depends on the quantities of inputs used, as well as their corresponding yields.

\begin{equation}
0.10*\text{CR}_1 + 0.15*\text{CR}_2 = \text{LN}
\tag{4.1}
\end{equation}

\begin{equation}
0.20*\text{CR}_1 + 0.25*\text{CR}_2 = \text{MN}
\tag{4.2}
\end{equation}

\begin{equation}
0.20*\text{CR}_1 + 0.18*\text{CR}_2 = \text{HN}
\tag{4.3}
\end{equation}

\begin{equation}
0.12*\text{CR}_1 + 0.08*\text{CR}_2 = \text{LO}
\tag{4.4}
\end{equation}

\begin{equation}
0.20*\text{CR}_1 + 0.19*\text{CR}_2 = \text{HO}
\tag{4.5}
\end{equation}

\begin{equation}
0.13*\text{CR}_1 + 0.12*\text{CR}_2 = \text{R}
\tag{4.6}
\end{equation}

\begin{equation}
0.60*\text{LNRG} + 0.52*\text{MNRG} + 0.45*\text{HNRG} = \text{RG}
\tag{4.7}
\end{equation}

\begin{equation}
0.68*\text{LOCGO} + 0.75*\text{HOCGO} = \text{CO}
\tag{4.8}
\end{equation}

\begin{equation}
0.28*\text{LOCGO} + 0.20*\text{HOCGO} = \text{CG}
\tag{4.9}
\end{equation}

\begin{equation}
0.50*\text{RLBO} = \text{LBO}
\tag{4.10}
\end{equation}

\begin{equation}
\text{LNPMF} + \text{MNPMF} + \text{HNPMF} + \text{RGPMF} + \text{CGPMF} = \text{PMF}
\tag{4.11}
\end{equation}

\begin{equation}
\text{LNRMF} + \text{MNRMF} + \text{HNRMF} + \text{RGRMF} + \text{CGRMF} = \text{RMF}
\tag{4.12}
\end{equation}

\begin{equation}
\text{LOJF} + \text{HOJF} + \text{COJF} + \text{RJF} = \text{JF}
\tag{4.13}
\end{equation}

- **Mass Conservation:** The number of barrels used must be equal to the number of barrels available.

\begin{equation}
\text{LNRG} + \text{LNPMF} + \text{LNRMF} = \text{LN}
\tag{5.1}
\end{equation}

\begin{equation}
\text{MNRG} + \text{MNPMF} + \text{MNRMF} = \text{MN}
\tag{5.2}
\end{equation}

\begin{equation}
\text{HNRG} + \text{HNPMF} + \text{HNRMF} = \text{HN}
\tag{5.3}
\end{equation}

\begin{equation}
\text{LOCGO} + \text{LOJF} + 0.55*\text{FO} = \text{LO}
\tag{5.4}
\end{equation}

\begin{equation}
\text{HOCGO} + \text{HOJF} + 0.17*\text{FO} = \text{HO}
\tag{5.5}
\end{equation}

\begin{equation}
\text{COJF} + 0.22*\text{FO} = \text{CO}
\tag{5.6}
\end{equation}

\begin{equation}
\text{RLBO} + \text{RJF} + 0.0555*\text{FO} = \text{R}
\tag{5.7}
\end{equation}

\begin{equation}
\text{CGPMF} + \text{CGRMF} = \text{CG}
\tag{5.8}
\end{equation}

\begin{equation}
\text{RGPMF} + \text{RGRMF} = \text{RG}
\tag{5.9}
\end{equation}

- **Premium-to-Regular Proportion:** The production ratio between premium and regular gasoline must satisfy the minimum requirement.

\begin{equation}
\text{PMF} \geq 0.40*\text{RMF}
\tag{6}
\end{equation}

- **Octane Tolerance:** The octane rating of each type of gasoline cannot drop below the lower tolerance.

\begin{equation}
90*\text{LNPMF} + 80*\text{MNPMF} + 70*\text{HNPMF} + 115*\text{RGPMF} + 105*\text{CGPMF} \geq 94*\text{PMF}
\tag{7.1}
\end{equation}

\begin{equation}
90*\text{LNRMF} + 80*\text{MNRMF} + 70*\text{HNRMF} + 115*\text{RGRMF} + 105*\text{CGRMF} \geq 84*\text{PMF}
\tag{7.2}
\end{equation}

- **Vapor-Pressure Tolerance:** The vapor pressure of jet fuel cannot drop below the lower tolerance.

\begin{equation}
1.0*\text{LOJF} + 0.6*\text{HOJF} + 1.5*\text{COJF} + 0.05*\text{RJF} \leq 1.0*\text{JF}
\tag{8}
\end{equation}

---
## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
import numpy as np

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input Data
We define all the input data of the model.

```python
# Parameters

crude_numbers = range(1,2+1)
petrols = ["Premium_fuel", "Regular_fuel"]
end_product_names = ["Premium_fuel", "Regular_fuel", "Jet_fuel", "Fuel_oil", "Lube_oil"]
distillation_products_names = ["Light_naphtha", "Medium_naphtha", "Heavy_naphtha",
                               "Light_oil", "Heavy_oil", "Residuum"]
naphthas = ["Light_naphtha", "Medium_naphtha", "Heavy_naphtha"]
intermediate_oils = ["Light_oil", "Heavy_oil"]
cracking_products_names = ["Cracked_gasoline", "Cracked_oil"]
used_for_motor_fuel_names = ["Light_naphtha", "Medium_naphtha", "Heavy_naphtha",
                             "Reformed_gasoline", "Cracked_gasoline"]
used_for_jet_fuel_names = ["Light_oil", "Heavy_oil", "Residuum", "Cracked_oil"]

buy_limit = {1:20000, 2:30000}
lbo_min = 500
lbo_max = 1000

distill_cap = 45000
reform_cap = 10000
crack_cap = 8000

distillation_splitting_coefficients = {"Light_naphtha": (0.1, 0.15),
                          "Medium_naphtha": (0.2, 0.25),
                         "Heavy_naphtha": (0.2, 0.18),
                         "Light_oil": (0.12, 0.08),
                         "Heavy_oil": (0.2, 0.19),
                         "Residuum": (0.13, 0.12)}

cracking_splitting_coefficients = {("Light_oil","Cracked_oil"): 0.68,
                                   ("Heavy_oil","Cracked_oil"): 0.75,
                                   ("Light_oil","Cracked_gasoline"): 0.28,
                                   ("Heavy_oil","Cracked_gasoline"): 0.2}

reforming_splitting_coefficients = {"Light_naphtha": 0.6, "Medium_naphtha":0.52, "Heavy_naphtha":0.45}
end_product_profit = {"Premium_fuel":7, "Regular_fuel":6, "Jet_fuel":4, "Fuel_oil":3.5, "Lube_oil":1.5}
blending_coefficients = {"Light_oil": 0.55, "Heavy_oil": 0.17, "Cracked_oil": 0.22, "Residuum": 0.055}

lube_oil_factor = 0.5
pmf_rmf_ratio = 0.4

octance_number_coefficients = {
    "Light_naphtha":90,
    "Medium_naphtha":80,
    "Heavy_naphtha":70,
    "Reformed_gasoline":115,
    "Cracked_gasoline":105,
}
octance_number_fuel = {"Premium_fuel": 94,"Regular_fuel": 84}

vapor_pressure_constants = [0.6, 1.5, 0.05]
```

## Model Deployment
We create a model and the variables.

```python
refinery = gp.Model('Refinery_Optimization')

# Variables
crudes = refinery.addVars(crude_numbers, ub=buy_limit, name="cr")
end_products = refinery.addVars(end_product_names, name="end_prod")
end_products["Lube_oil"].lb= lbo_min
end_products["Lube_oil"].ub= lbo_max
distillation_products = refinery.addVars(distillation_products_names, name="dist_prod")
reform_usage = refinery.addVars(naphthas, name="napthas_to_reformed_gasoline")
reformed_gasoline = refinery.addVar(name="reformed_gasoline")
cracking_usage = refinery.addVars(intermediate_oils,name="intermediate_oils_to_cracked_gasoline")
cracking_products = refinery.addVars(cracking_products_names,  name="cracking_prods")
used_for_regular_motor_fuel = refinery.addVars(used_for_motor_fuel_names, name="motor_fuel_to_regular_motor_fuel")
used_for_premium_motor_fuel = refinery.addVars(used_for_motor_fuel_names, name="motot_fuel_to_premium_motor_fuel")
used_for_jet_fuel = refinery.addVars(used_for_jet_fuel_names, name="jet_fuel")
used_for_lube_oil = refinery.addVar(vtype=GRB.CONTINUOUS,name="residuum_used_for_lube_oil")
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-

Next, we insert the constraints.

The distillation capacity constraint is:

```python
# 1. Distillation capacity
DistillationCap = refinery.addConstr(crudes.sum() <= distill_cap, "Distill_cap")
```

The reforming capacity constraint is:

```python
# 2. Reforming capacity
ReformingCap = refinery.addConstr(reform_usage.sum() <= reform_cap, "Reform_cap")
```

The cracking capacity constraint is:

```python
# 3. Cracking capacity
CrackingCap = refinery.addConstr(cracking_usage.sum() <= crack_cap, "Crack_cap")
```

The quantity of distillation products produced depends on the quantity of crude oil used, taking into account the way in which each crude splits under distillation. This gives:

```python
# 4.1-4.6 Yield (Crude oil products)
YieldCrudeOil = refinery.addConstrs((gp.quicksum(distillation_splitting_coefficients[dpn][crude-1]*crudes[crude] for crude in crudes)
                  == distillation_products[dpn] for dpn in distillation_products_names), "Splitting_distillation")
```

The quantity of reformed gasoline produced depends on the quantities of naphthas used in the reforming process. This gives the constraint:

```python
# 4.7 Yield (Reforming of Naphthas)
YieldNaphthas = refinery.addConstr(reform_usage.prod(reforming_splitting_coefficients) == reformed_gasoline, "Splitting_reforming")
```

The quantities of cracked oil and cracked gasoline produced depend on the quantities of light and heavy oil used. This gives the constraints:

```python
# 4.8-4.9 Yield (Cracking of oils)
YieldCrackingOil = refinery.addConstrs((gp.quicksum(cracking_splitting_coefficients[oil, crack_prod]*cracking_usage[oil]
                           for oil in intermediate_oils) == cracking_products[crack_prod]
                  for crack_prod in cracking_products_names),
                 name="Splitting_cracking")
```

The quantity of lube-oil produced (and sold) is 0.5 times the quantity of residuum used. This gives:

```python
# 4.10 Yield (Lube oil)
YieldLubeOil = refinery.addConstr(lube_oil_factor*used_for_lube_oil == end_products["Lube_oil"],
                "Splitting_lube_oil")
```

The quantity of motor fuels and jet fuel produced is equal to the total quantity of their ingredients. This gives the constraints:

```python
# 4.11 Yield (Premium gasoline)
YieldPremium = refinery.addConstr(used_for_premium_motor_fuel.sum() == end_products["Premium_fuel"], "Blending_premium_fuel")

# 4.12 Yield (Regular gasoline)
YieldRegular = refinery.addConstr(used_for_regular_motor_fuel.sum() == end_products["Regular_fuel"], "Blending_regular_fuel")

# 4.13 Yield (Jet fuel)
YieldJetFuel = refinery.addConstr(used_for_jet_fuel.sum() == end_products["Jet_fuel"], "Continuity_jet_fuel")
```

The quantities of naphthas used for reforming and blending are equal to the quantities available. This gives:

```python
# 5.1-5.3 Mass conservation (Naphthas)
MassBalNaphthas = refinery.addConstrs((reform_usage[naphtha] +
                    used_for_regular_motor_fuel[naphtha] +
                    used_for_premium_motor_fuel[naphtha] ==
                    distillation_products[naphtha] for naphtha in naphthas), "Continuity_napththa")
```

For the blending of fuel oil, the proportion of light oil/heavy oil/ cracked oil/ residuum is fixed. Therefore, separate variables have not been introduced for this proportion as it is determined by the variables. This gives:

```python
# 5.4 Mass Conservation (Light oil)
MassBalLightOil = refinery.addConstr(cracking_usage["Light_oil"]+
                used_for_jet_fuel["Light_oil"]+
                blending_coefficients["Light_oil"]*end_products["Fuel_oil"] ==
                distillation_products["Light_oil"], "Fixed_proportion_light_oil_for_blending")

# 5.5 Mass Conservation (Heavy oil)
MassBalHeavyOil = refinery.addConstr(cracking_usage["Heavy_oil"]+
                used_for_jet_fuel["Heavy_oil"]+
                blending_coefficients["Heavy_oil"]*end_products["Fuel_oil"] ==
                distillation_products["Heavy_oil"], "Fixed_proportion_heavy_oil_for_blending")

# 5.6 Mass Conservation (Cracked oil)
MassBalCrackedOil = refinery.addConstr(used_for_jet_fuel["Cracked_oil"]+
                blending_coefficients["Cracked_oil"]*end_products["Fuel_oil"] ==
                cracking_products["Cracked_oil"], "Fixed_proportion_cracked_oil_for_blending")

# 5.7 Mass Conservation (Residuum)
MassBalResiduum = refinery.addConstr(used_for_lube_oil +
                used_for_jet_fuel["Residuum"]+
                blending_coefficients["Residuum"]*end_products["Fuel_oil"] ==
                distillation_products["Residuum"], "Fixed_proportion_residuum_for_blending")

# 5.8 Mass conservation (Cracked gasoline)
MassBalCrackedGas = refinery.addConstr(used_for_regular_motor_fuel["Cracked_gasoline"] +
                used_for_premium_motor_fuel["Cracked_gasoline"] ==
                cracking_products["Cracked_gasoline"], "Continuity_cracked_gasoline")

# 5.9 Mass conservation (Reformed gasoline)
MassBalReformedGas = refinery.addConstr(used_for_regular_motor_fuel["Reformed_gasoline"] +
                used_for_premium_motor_fuel["Reformed_gasoline"] ==
                reformed_gasoline, "Continuity_reformed_gasoline")
```

Premium motor fuel production must be at least 40% of regular motor fuel production, giving:

```python
# 7. Premium-to-regular proportion
Premium2Regular = refinery.addConstr(end_products["Premium_fuel"] >= pmf_rmf_ratio*end_products["Regular_fuel"],
                "Prem2reg_prop")
```

It is necessary to stipulate that the octane number of premium motor (regular motor) fuel does not drop below 94 (84). This is done by the constraint:

```python
# 8.1-8.2 Octane tolerance
OctaneRegular = refinery.addConstr(used_for_regular_motor_fuel.prod(octance_number_coefficients) >=
                octance_number_fuel["Regular_fuel"] * end_products["Regular_fuel"],
                "Octane_tol_regular_fuel")

OctanePremium = refinery.addConstr(used_for_premium_motor_fuel.prod(octance_number_coefficients) >=
                octance_number_fuel["Premium_fuel"] * end_products["Premium_fuel"],
                "Octane_tol_premium_fuel")
```

For jet fuel, we have the constraint imposed by vapor pressure:

```python
# 9. Vapor-pressure tolerance
VaporPressure = refinery.addConstr(used_for_jet_fuel["Light_oil"] +
                vapor_pressure_constants[0]*used_for_jet_fuel["Heavy_oil"] +
                vapor_pressure_constants[1]*used_for_jet_fuel["Cracked_oil"] +
                vapor_pressure_constants[2]*used_for_jet_fuel["Residuum"] <= end_products["Jet_fuel"],
                "Vapor_pressure_tol")
```

This model has 29 constraints together with simple bounds on three variables.

A note should be made concerning the blending of fuel oil where the ingredients (light, heavy, and cracked oil and residuum) are used in fixed proportions. It might be preferable to think of the production of fuel oil as an activity. It is common in the oil industry to think in terms of activities rather than quantities. In Section 3.4 of the H.P. Williams book, model formulations are discussed where activities represent the extreme modes of operation of a process. In this example, we have a special case of a process with one mode of operation. The level of this activity then fixes the ratios of the ingredients, in a case such as this, automatically.

The only variables involving a profit (or cost) are the final products. This gives an objective (in dollars) to be maximized of:

```python
# 0. Profit
refinery.setObjective(end_products.prod(end_product_profit), GRB.MAXIMIZE)
```

Next, the optimization process starts and Gurobi finds the optimal solution.

```python
refinery.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 29 rows, 36 columns and 106 nonzeros
    Model fingerprint: 0xe30699e6
    Coefficient statistics:
      Matrix range     [5e-02, 1e+02]
      Objective range  [2e+00, 7e+00]
      Bounds range     [5e+02, 3e+04]
      RHS range        [8e+03, 5e+04]
    Presolve removed 13 rows and 14 columns
    Presolve time: 0.01s
    Presolved: 16 rows, 22 columns, 72 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    1.1887574e+06   6.045565e+04   0.000000e+00      0s
          14    2.1136513e+05   0.000000e+00   0.000000e+00      0s

    Solved in 14 iterations and 0.01 seconds
    Optimal objective  2.113651348e+05

---
## Analysis

The optimal solution results in a profit of $\$211,365.13$. The optimal values of the variables are given below:

```python
for var in refinery.getVars():
    if abs(var.x) > 1e-6:
        print("{0} = {1}".format(var.varName, np.round(var.x, 2)))
```

    cr[1] = 15000.0
    cr[2] = 30000.0
    end_prod[Premium_fuel] = 6817.78
    end_prod[Regular_fuel] = 17044.45
    end_prod[Jet_fuel] = 15156.0
    end_prod[Lube_oil] = 500.0
    dist_prod[Light_naphtha] = 6000.0
    dist_prod[Medium_naphtha] = 10500.0
    dist_prod[Heavy_naphtha] = 8400.0
    dist_prod[Light_oil] = 4200.0
    dist_prod[Heavy_oil] = 8700.0
    dist_prod[Residuum] = 5550.0
    napthas_to_reformed_gasoline[Heavy_naphtha] = 5406.86
    reformed_gasoline = 2433.09
    intermediate_oils_to_cracked_gasoline[Light_oil] = 4200.0
    intermediate_oils_to_cracked_gasoline[Heavy_oil] = 3800.0
    cracking_prods[Cracked_gasoline] = 1936.0
    cracking_prods[Cracked_oil] = 5706.0
    motor_fuel_to_regular_motor_fuel[Light_naphtha] = 273.07
    motor_fuel_to_regular_motor_fuel[Medium_naphtha] = 10500.0
    motor_fuel_to_regular_motor_fuel[Heavy_naphtha] = 2993.14
    motor_fuel_to_regular_motor_fuel[Reformed_gasoline] = 1342.24
    motor_fuel_to_regular_motor_fuel[Cracked_gasoline] = 1936.0
    motot_fuel_to_premium_motor_fuel[Light_naphtha] = 5726.93
    motot_fuel_to_premium_motor_fuel[Reformed_gasoline] = 1090.84
    jet_fuel[Heavy_oil] = 4900.0
    jet_fuel[Residuum] = 4550.0
    jet_fuel[Cracked_oil] = 5706.0
    residuum_used_for_lube_oil = 1000.0

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```


```python
# Aquí van las librerías y funciones que va a usar

import sympy as sym
import numpy as np
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt

sym.init_printing()

## Puede que requiera incorporar más librerías o definir nuevas funciones.  Hágalo a continuación

```

# Series de Fourier: Tarea

Nombres: **Ponga aquí sus nombres completos**

## Caso A

Use la siguiente función para definir un periodo de una señal periódica con periodo $T=1$:

$$f = \frac{1}{T=1}$$
$$ \omega = 2 \pi f = \frac{2 \pi}{T}$$

\begin{equation}
y_1(t) = \begin{cases}
-1 &  0 \leq t < 0.5 \\
1 &  0.5 \leq t < 1
\end{cases}
\end{equation}

```python
## Defina aquí y1

```

Varíe la cantidad de componentes que reconstruyen la función mediante series de Fourier y visualice los espectros y las reconstrucciones.

```python
## Escriba su código. Use más celdas si es necesario
```

## Caso B

Use la siguiente función para definir un periodo de una señal periódica con periodo $T=1$:

\begin{equation}
y_2(t) = t
\end{equation}

```python
## Defina aquí y2

```

Varíe la cantidad de componentes que reconstruyen la función mediante series de Fourier y visualice los espectros y las reconstrucciones.

```python
## Escriba su código. Use más celdas si es necesario
```

## Caso C

Use la siguiente función para definir un periodo de una señal periódica con periodo $T=1$:

\begin{equation}
y_3(t) = 3 sin(2 \pi t) +2
\end{equation}

```python
## Defina aquí y3

```

Varíe la cantidad de componentes que reconstruyen la función mediante series de Fourier y visualice los espectros y las reconstrucciones.

```python
## Escriba su código. Use más celdas si es necesario
```

## Análisis comparativo

Con base en los resultados anteriores, realice un análisis comparativo y escriba sus observaciones

- Escriba aquí sus respuestas
- Puede usar viñetas

## Conclusiones

Escriba sus conclusiones

- Use viñetas

```python

```


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 1. Functions and Coordinate Sets

Chapter 1 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

**Learning Objectives**

By the end of this session, you should be able to:
- Plot functions in Python with `sympy`, `matplotlib` and `numpy`
- Understand the exponential function $e^x$ and other common functions
- Understand odd and even functions

### 1.a Functions

As a reminder, a function maps numbers (e.g., 'x' values) to other numbers (e.g., 'y' values):

Obviously, it's nice to plot functions. For simple functions, we can use the `sympy` package (symbolic math).

For more complicated things, we'll use `matplotlib` later.

```python
# import sympy and start a session
from sympy import init_session
init_session()
```

Sympy just defined a bunch of symbols we can use in our plots (e.g., x, y, z, t, etc.) and set up fancy formula printing. We can now import and call the `plot` function:

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

from sympy.plotting import plot
```

```python
plot(sin(x))
```

**Nice !** We have a pretty easy way to plot functions in our Jupyter notebooks - we can learn to tweak the plots later (e.g., axes labels, etc.), but this is good enough for now.

We can plot multiple functions in one graph.

```python
p1 = plot(2*x - 2, 3-x)
```

As a reminder, the slope of the lines can be calculated, e.g.
$$ slope = \frac{\Delta y}{\Delta x} = \frac{6 - 0}{4 - 1} = 2$$

### 1.a.1 Quadratic Functions

Quadratic equations take the form:
$$ y(x) = ax^2 + bx + c$$

For example:

```python
plot(-x**2 + 2*x - 1, xlim=(-4, 4))  # set the x-limit  -4 to +4
plot(x**2 + 2*x + 0.75, line_color='red')
plot(x**2 - 2*x + 2, line_color='gold') # can be any HTML color name
```

You probably remember that we can solve quadratic equations:
$$ x = \frac{-b \pm \sqrt{b^2-4ac}}{2a}$$

The $\sqrt{b^2-4ac}$ part is important, since it tells us if we get real numbers for our solutions when $y(x) = 0$

### Chemical Equlibrium example:

Let's consider a chemistry example from the textbook:
$$\require{mhchem}$$
$$\ce{PCl5 (g) -> PCl3 (g) + Cl2 (g)}$$

So we can write an equilibrium constant:

$$K_P = \frac{P_{PCl_3}P_{Cl_2}}{P_{PCl_5}}$$

If we start with 1.0 atm pressure, then we get:

$$K_P = \frac{[x] \; [x]}{[1.0 - x]}$$

Then:

$$x^2 + K_P x − K_p = 0$$

```python
Kp = 1.0 # arbitrary number to allow us to solve this numerically

# solve a quadratic = (-b +/- sqrt(b**2 - 4ac)) / 2a
x1 = (-Kp + np.sqrt(Kp**2 - 4*1*(-Kp))) / (2*1)
x2 = (-Kp - np.sqrt(Kp**2 - 4*1*(-Kp))) / (2*1)

print(round(x1,2))
print(round(x2,2)) # too many sig figs again...

print(round(1 - x1, 2))
```

Obviously, we can't have negative numbers for pressures, so we can ignore one of these two solutions. Algebraically, we see:

$$
P_{\mathrm{Cl}_{2}}=P_{\mathrm{PCl}_{3}}=\frac{1}{2} \mathrm{K}_{\mathrm{p}}\left[1+\sqrt{1+4 / \mathrm{K}_{\mathrm{p}}}\right]
$$

And of course:
$$
P_{\mathrm{PCl}_{5}}=1.0- P_{\mathrm{Cl}_{2}}
$$

### 1.a.2 Beyond Quadratics

Obviously, there are higher-order polynomial functions. We usually note the type or category of the polynomial by the highest-order exponent:

- $x$ - linear
- $x^2$ - quadratic
- $x^3$ - cubic
- $x^4$ - quartic
- $x^5$ - quintic

There are analytical ways to solve linear, quadratic, cubic, and quartic functions. It's been proven that [quintic](https://en.wikipedia.org/wiki/Quintic_function) equations cannot always be solved.

```python
plot(4*(x-1)**2, line_color='red') # quadratic
plot(-x**3, line_color='grey') # cubic
plot((x**2 - 1)**2, line_color='blue') # quartic
```

### 1.a.3 Odd and Even Functions

Think about the function $y = -x^3$ above. For any value $x$, we can see that:
$$y(x) = -y(-x)$$

In other words, the function is inverted on opposite side of $x = 0$. This is an example of an *odd function*. In contrast, $y = (x^2 - 1)^2$ is an *even function* where $y(x) = y(-x)$. It has mirror symmetry around $x = 0$.

```python
plot(sin(x))
```

Is $\sin(x)$ odd or even? - ODD

```python
plot(cos(x))
```

What about $cos(x)$? Is it an odd or even function? - EVEN

**Note:** Some functions are neither, for example $y = 4(x-1)^2$ is not odd or even because it has *no* symmetry at all around $x = 0$.

***Why does this matter?***

For one, it's easier to sketch a function by hand if you know that it's even or odd, since there's some symmetry around $x = 0$.

More importantly, some kinds of integrals can be massively simplified if you know the function is odd or even. For example, consider;

$$\int_{-\infty}^{\infty} \sin x dx = 0$$

Since we know $\sin x$ is an odd function, *whatever* the integral from 0 to $\infty$ it will be exactly negated by the integral from $-\infty$ to 0.

We'll discuss this further when we get to integrals.

### 1.a.4 Plot the function

Lots of people / texts will tell you to draw the function. When I took math courses, they spent a lot of time on sketching functions. It's 2020, and I carry a powerful computer in my pocket (i.e., my iPhone). As you can see, with a small amount of Python, we can plot functions in a web browser.

To borrow from an ad campaign, and one of my favorite college professors, here's

## Hutchison's First Law: Just Plot It!

This will tell you things like maximum or minimum values, whether it's continuous, how it behaves (e.g., is it only defined on a range of values), etc. Hopefully if you're not comfortable plotting functions in Python, you will be soon.

### 1.a.5 Useful functions:

**Exponentials.**

In many cases, growth can be much faster than linear, quadratic, etc. (e.g. COVID):
$$y = e^x$$

```python
exp(x)
```

There's also exponential decay (e.g, radioactive elements):
$$y = e^{-x}$$

```python
plot(exp(x),
     exp(-x),
     ylim=(0, 12))
```

Note that $e^{2.3} \approx 10$. We can find the exact value with the $log()$ function, which is the inverse of exponentials.

```python
import numpy as np

np.log(10)
```

**Logarithms.**

We will dicuss logarithms in more detail soon. Suffice to say, it serves as the inverse of exponential:

$$ \ln(\exp(x)) = \exp(\ln(x)) = x$$

One recent example for the need of logarithms was to visualize exponential growth (e.g., COVID infections):

```python
plot(exp(x),
    30 + 0.5*exp(x),
     yscale="log")
```

Notice that while we plotted two exponential functions, they appear as straight lines on a log scale plot. Similarly, non-exponential scaling (e.g., the start of the orange line for $30 + 0.5\exp(x)$) appears 'flat' - thus the idea of "flattening the curve" in a pandemic. Note that the original spread was exponential (a diagonal line) before initial quarantine efforts slowed progress to linear spread.

(Data is from the [Johns Hopkins COVID19 Dashboard](https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6) via their [GitHub archive](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series).)

**Sinusoid / Periodic.**

Both sine and cosine functions have periodicity as we saw above, including multiple maxima and minima, etc.

```python
plot(sin(x), line_color='blue')
plot(cos(x), line_color='gold')
```

**Compound Functions.**

Many physical behaviors can be modeled by a product of functions, e.g.:

$$f(t) = exp(-t/\tau) cos(2\pi\nu t)$$

This is an example for a "*damped*" spring - note that while the $\cos(x)$ is fully periodic, the composite will decrease in amplitude over time because of the $\exp(-t)$ term.

```python
plot(cos(2*pi*x), line_color='black')
plot(exp(-x), line_color='blue')
```

```python
plot(x, 2*exp(-x)*cos(2*pi*x), line_color='red')
```

**Gaussians.**

Another common function across science is the so-called [Gaussian function](https://en.wikipedia.org/wiki/Gaussian_function) which emerges from the so-called [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution) or so-called "bell curve" for distributions of large numbers (e.g., standardized test scores, speeds of molecules in a gas, etc.). We will discuss this more later in the course where it shows up frequently in statistics and probability.

$$p(x)=\exp \left(-\frac{1}{2 \sigma^{2}} (x-x_0)^{2}\right)$$

```python
# first Gaussian, sigma = 2.0, mean (x0) = 4.0
sigma = 2.0
x0 = 4.0
plot(exp(-1.0/(2.0*sigma**2) * (x - x0)**2), line_color='blue')
```

```python
# plot a narrower Gaussian with sigma = 1.0 and same x0
sigma2 = 1.0
x0 = 4.0
plot(exp(-1.0/(2.0*sigma2**2) * (x - x0)**2), line_color='red')
```

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000