# Validating Runge Kutta Butcher tables using Truncated Taylor Series
## Authors: Zach Etienne & Brandon Clark

## This tutorial notebook is designed to validate the Butcher tables contained within the Butcher dictionary constructed in the [RK Butcher Table Dictionary](Tutorial-RK_Butcher_Table_Dictionary.ipynb) NRPy+ module.

### NRPy+ Source Code for this module:
* [MoLtimestepping/RK_Butcher_Table_Validation.py](../edit/MoLtimestepping/RK_Butcher_Table_Validation.py) Stores the `Validate` function for calidating convergence orders for Runge Kutta methods
* [MoLtimestepping/RK_Butcher_Table_Dictionary.py](../edit/MoLtimestepping/RK_Butcher_Table_Dictionary.py) [\[**tutorial**\]](Tutorial-RK_Butcher_Table_Dictionary.ipynb) Accesses the Butcher table dictionary `Butcher_dict` for known explicit Runge Kutta methods

## Introduction:

Starting with the ODE (ordinary differential equation) initial value problem:
$$
y'(t) = f(y,t)\ \ \ y\left(t=0\right)=y_0,
$$
for various choices of $f(y,t)$, this module validates the Runge Kutta (RK) methods coded in [RK_Butcher_Table_Dictionary.py](../edit/MoLtimestepping/RK_Butcher_Table_Dictionary.py) [**tutorial notebook**](Tutorial-RK_Butcher_Table_Dictionary.ipynb) as follows:

Given $y_0$, and a smooth $f(y,t)$, all explicit RK methods provide an estimate for $y_1 = y\left(\Delta t\right)$, with an error term that is proportional to $\left(\Delta t\right)^m$, where $m$ is an integer typically greater than zero. This error term corresponds to the *local* truncation error. For RK4, for example, while the *total accumulated truncation error* (i.e., the accumulated error at a fixed final time $t_f$) is proportional to $\left(\Delta t\right)^4$, the *local* truncation error (i.e., the error after one arbitrarily chosen timestep $\Delta t$) is proportional to $\left(\Delta t\right)^5$.

If the exact solution $y(t)$ is known as a closed-form expression, then $y\left(\Delta t\right)$ can be *separately* written as a Taylor expansion about $y(t=0)$:

$$
y\left(\Delta t\right) = \sum_{n=0}^\infty \frac{y^{(n)}(t=0)}{n!} \left(\Delta t\right)^n,
$$
where $y^{(n)}(t=0)$ is the $n$th derivative of $y(t)$ evaluated at $t=0$.

The above expression will be known exactly. Further if one chooses a numerical value for $y_0$ *and leaves $\Delta t$ unspecified*, any explicit RK method will provide an estimate for $y\left(\Delta t\right)$ of the form

$$
y\left(\Delta t\right) = \sum_{n=0}^\infty a_n \left(\Delta t\right)^n,
$$
where $a_n$ *must* match the Taylor expansion of the *exact* solution at least up to and including terms proportional to $\left(\Delta t\right)^m$, where $m$ is the order of the local truncation error. If this is *not* the case, then the Butcher table was almost certainly *not* typed correctly.

Therefore, comparing the numerical result with unspecified $\Delta t$ against the exact Taylor series provides a convenient (though not perfectly robust) means to verify that the Butcher table for a given RK method was typed correctly. Multiple typos in the Butcher tables were found using this approach.

**Example from Z. Etienne's MATH 521 (Numerical Analysis) lecture notes:**

Consider the ODE
$$
y' = y - 2 t e^{-2t},\quad y(0)=y(t_0)=0.
$$

* Solve this ODE exactly, then Taylor expand the solution about $t=0$ to
approximate the solution at $y(t=\Delta t)$ to fifth order in $\Delta
t$.
* Next solve this ODE using Heun's method (second order in total accumulated truncation error, third order in local truncation error) {\it by hand} with a step size of
$\Delta t$ to find $y(\Delta t)$. Confirm that the solution obtained
when using Heun's method has an error term that is at worst
$\mathcal{O}\left((\Delta t)^3\right)$. If the dominant error is
proportional to a higher power of $\Delta t$, explain the discrepancy.

* Finally solve this ODE using the Ralston method {\it by hand}
  with a step size of $\Delta t$ to find $y(\Delta t)$. Is the
  coefficient on the dominant error term closer to the exact solution
  than Heun's method?

We can solve this equation via the method of integrating factors,
which states that ODEs of the form:
$$
y'(t) + p(t) y(t) = g(t)
$$
are solved via
$$
y(t) = \frac{1}{\mu(t)} \left[ \int \mu(s) g(s) ds + c \right],
$$
where the integrating factor $\mu(t)$ is given by
$$
\mu(t) = \exp\left(\int p(t) dt\right)
$$

Here, $p(t)=-1$ and $g(t) = - 2 t e^{-2t}$. Then
\beq
\mu(t) = \exp\left(-\int dt\right) = e^{-t+c} = k e^{-t}
\eeq
and
\begin{align}
y(t) &= e^t/k  \left[ \int k e^{-s} (- 2 s e^{-2s}) ds + c \right] = -2 e^t \left[ \int s e^{-3s} ds + c' \right] \\
&= -2 e^t \left[ e^{-3 t} \left(-\frac{t}{3}-\frac{1}{9}\right) + c' \right] = -2 e^{-2t} \left(-\frac{t}{3}-\frac{1}{9}\right) -2 c' e^t \\
&= e^{-2t} \left(2\frac{t}{3}+\frac{2}{9}\right) + c'' e^t \\
\end{align}

If $y(0)=0$ then we can compute the integration constant $c''$, and
$y(t)$ becomes
$$
y(t) = \frac{2}{9} e^{-2 t} \left(3 t + 1 - e^{3 t}\right).
$$

The Taylor Series expansion of the exact solution about $t=0$
evaluated at $y(\Delta t)$ yields
$$
y(\Delta t) = -(\Delta t)^2+(\Delta t)^3-\frac{3 (\Delta t)^4}{4}+\frac{23 (\Delta
  t)^5}{60}-\frac{19 (\Delta t)^6}{120}+O\left((\Delta t)^7\right).
$$

Next we evaluate $y(\Delta t)$ using Heun's method. We know $y(0)=y_0=0$ and
$f(y,t)=y - 2 t e^{-2t}$, so
\begin{align}
k_1 &= \Delta t f(y(0),0) \\
    &= \Delta t \times 0 \\
    &= 0 \\
k_2 &= \Delta t f(y(0)+k_1,0+\Delta t) \\
   &= \Delta t f(y(0)+0,0+\Delta t) \\
   &= \Delta t (-2 \Delta t e^{-2\Delta t}) \\
   &= -2 (\Delta t)^2 e^{-2\Delta t} \\
y(\Delta t) &= y_0 + \frac{1}{2} (k_1 + k_2) + \mathcal{O}\left((\Delta t)^3\right) \\
&= 0 - (\Delta t)^2 e^{-2\Delta t} \\
&= - (\Delta t)^2 ( 1 - 2 \Delta t + 2 (\Delta t)^2 + ...) \\
&= - (\Delta t)^2 + 2 (\Delta t)^3 + \mathcal{O}\left((\Delta t)^4\right).
\end{align}

Thus the coefficient on the $(\Delta t)^3$ term is wrong, but
this is completely consistent with the fact that our stepping
scheme is only third-order accurate in $\Delta t$.

In the below approach, the RK result is subtracted from the exact Taylor series result, as a check to determine whether the RK Butcher table was coded correctly; if it was not, then the odds are good that the RK results will not match to the expected local truncation error order. Multiple $f(y,t)$ are coded below to improve the robustness of this test.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize needed Python/NRPy+ modules
1. [Step 2](#table_validate) Validate Convergence Order of Butcher Tables
    1. [Step 2.a](#rhs): Defining the right-hand side of the ODE
    1. [Step 2.b](#validfunc): Defining a Validation Function
    1. [Step 2.c](#rkvalid): Validating RK Methods against ODEs
1. [Step 3](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize needed Python/NRPy+ modules [Back to [top](#toc)\]
$$\label{initializenrpy}$$

Let's start by importing all the needed modules from Python/NRPy+:

```python
import sympy as sp
import NRPy_param_funcs as par
import numpy as np
from MoLtimestepping.RK_Butcher_Table_Dictionary import Butcher_dict
```

# Step 2:  Validate Convergence Order of Butcher Tables [Back to [top](#toc)\]
$$\label{table_validate}$$

Each Butcher table/Runge Kutta method is tested by solving an ODE. Comparing the Taylor series expansions of the exact solution and the numerical solution as discussed in the **Introduction** above will confirm whether the method converges to the appropriate order.

## Step 2.a:  Defining the right-hand side of the ODE [Back to [top](#toc)\]
$$\label{rhs}$$

Consider the form of ODE $y'=f(y,t)$. The following begins to construct a dictionary `rhs_dict` of right-hand side functions for us to validate explicit Runge Kutta methods. The most up-to-date catlog of functions stored in `rhs_dict` can be found in the [RK_Butcher_Table_Validation.py](../edit/MoLtimestepping/RK_Butcher_Table_Validation.py) module.

```python
def fypt(y,t): # Yields expected convergence order for all cases
    #            except DP6 which converge to higher order (7, respectively)
    return y+t

def fy(y,t): # Yields expected convergence order for all cases
    return y

def feypt(y,t): # Yields expected convergence order for all cases
    return sp.exp(1.0*(y+t))

def ftpoly6(y,t): # Yields expected convergence order for all cases, L6 has 0 error
    return 2*t**6-389*t**5+15*t**4-22*t**3+81*t**2-t+42
rhs_dict = {'ypt':fypt, 'y':fy, 'eypt':feypt, 'tpoly6':ftpoly6}
```

## Step 2.b: Defining a Validation Function  [Back to [top](#toc)\]
$$\label{validfunc}$$

To validate each Butcher table we compare the exact solutions to ODEs with the numerical solutions using the Runge Kutta scheme built into each Butcher table. The following is a function that

1. Solves the ODE exactly,
2. Solves the ODE numericaly for a given Butcher table, and
3. Compares the two solutions and checks for the order of convergence by returning their difference.

The `Validate()` function inputs a specified `Butcher_key`, the starting guess solution and time `y_n`, `t_n` and the right-hand side of the ODE corresponding to a specified intial value problem, `rhs_key`.

```python
from MoLtimestepping.RK_Butcher_Table_Dictionary import Butcher_dict
def Validate(Butcher_key, yn, tn, rhs_key):
    # 1. First we solve the ODE exactly
    y = sp.Function('y')
    sol = sp.dsolve(sp.Eq(y(t).diff(t), rhs_dict[rhs_key](y(t), t)), y(t)).rhs
    constants = sp.solve([sol.subs(t,tn)-yn])
    exact = sol.subs(constants)

    # 2. Now we solve the ODE numerically using specified Butcher table

    # Access the requested Butcher table
    Butcher = Butcher_dict[Butcher_key][0]
    # Determine number of predictor-corrector steps
    L = len(Butcher)-1
    # Set a temporary array for update values
    k = np.zeros(L, dtype=object)
    # Initialize intermediate variable
    yhat = 0
    # Initialize the updated solution
    ynp1 = 0
    for i in range(L):
        #Initialize and approximate update for solution
        yhat = yn
        for j in range(i):
            # Update yhat for solution using a_ij Butcher table coefficients
            yhat += Butcher[i][j+1]*k[j]
            if Butcher_key == "DP8" or Butcher_key == "L6":
                yhat = 1.0*sp.N(yhat,20) # Otherwise the adding of fractions kills performance.
        # Determine the next corrector variable k_i using c_i Butcher table coefficients
        k[i] = dt*rhs_dict[rhs_key](yhat, tn + Butcher[i][0]*dt)
        # Update the solution at the next iteration ynp1 using Butcher table coefficients
        ynp1 += Butcher[L][i+1]*k[i]
    # Finish determining the solution for the next iteration
    ynp1 += yn

    # Determine the order of the RK method
    order = Butcher_dict[Butcher_key][1]+2
    # Produces Taylor series of exact solution at t=tn about t = 0 with the specified order
    exact_series = sp.series(exact.subs(t, dt),dt, 0, order)
    num_series = sp.series(ynp1, dt, 0, order)
    diff = exact_series-num_series
    return diff
```

## Step 2.c: Validating RK Methods against ODEs  [Back to [top](#toc)\]
$$\label{rkvalid}$$

The following makes use of the `Validate()` function above to demonstrate that each method within the Bucther table dictionary converges to the expected order for the given right-hand side expression.

```python
t, dt = sp.symbols('t dt')
# Set intial conditions
t0 = 0
y0 = 1
# Set RHS of ODE
function = 'ypt'# This can be changed, just be careful that the initial conditions are satisfied
for key,value in Butcher_dict.items():
    print("RK method: \""+str(key)+"\".")
    y = sp.Function('y')
    print(" When solving y'(t) = "+str(rhs_dict[function](y(t),t))+", y("+str(t0)+")="+str(y0)+",")
    local_truncation_order = list(value)[1]+1
    print(" the first nonzero term should have local truncation error proportional to O(dt^"+str(local_truncation_order)+") or a higher power of dt.")
    print("Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:")
    sp.pretty_print(Validate(key, y0, t0, function))
#     print("\n")
    print(" (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)\n")
```

    RK method: "Euler".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^2) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      2    ⎛  3⎞
    dt  + O⎝dt ⎠
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK2 Heun".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^3) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      3
    dt     ⎛  4⎞
    ─── + O⎝dt ⎠
     3
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK2 MP".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^3) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      3
    dt     ⎛  4⎞
    ─── + O⎝dt ⎠
     3
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK2 Ralston".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^3) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      3
    dt     ⎛  4⎞
    ─── + O⎝dt ⎠
     3
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK3".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^4) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      4
    dt     ⎛  5⎞
    ─── + O⎝dt ⎠
     12
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK3 Heun".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^4) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      4
    dt     ⎛  5⎞
    ─── + O⎝dt ⎠
     12
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK3 Ralston".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^4) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      4
    dt     ⎛  5⎞
    ─── + O⎝dt ⎠
     12
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "SSPRK3".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^4) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      4
    dt     ⎛  5⎞
    ─── + O⎝dt ⎠
     12
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "RK4".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^5) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      5
    dt     ⎛  6⎞
    ─── + O⎝dt ⎠
     60
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "DP5".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^6) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
        6
      dt      ⎛  7⎞
    - ──── + O⎝dt ⎠
      1800
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "DP5alt".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^6) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
         6
    13⋅dt     ⎛  7⎞
    ────── + O⎝dt ⎠
    231000
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "CK5".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^6) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
      6
    dt      ⎛  7⎞
    ──── + O⎝dt ⎠
    3600
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "DP6".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^7) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
     ⎛  8⎞
    O⎝dt ⎠
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "L6".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^7) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
                                  5                               6
    - 1.0587911840678754238e-22⋅dt  - 2.6469779601696885596e-23⋅dt  + 0.0013227513

                   7    ⎛  8⎞
    227513227513⋅dt  + O⎝dt ⎠
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

    RK method: "DP8".
     When solving y'(t) = t + y(t), y(0)=1,
     the first nonzero term should have local truncation error proportional to O(dt^9) or a higher power of dt.
    Subtracting the numerical result from the exact Taylor expansion, we find a local truncation error of:
                                                               2
    3.6854403535034607753e-18⋅dt + 5.8394451383711465375e-18⋅dt  + 3.7764963953332

                 3                              4                               5
    980617e-18⋅dt  + 9.542884942003761195e-19⋅dt  + 1.2718729098615353529e-19⋅dt

                                  6                               7
    + 3.9082629581905451582e-20⋅dt  + 4.8075737201581968464e-21⋅dt  + 5.1688448526

                    8                              9    ⎛  10⎞
    907316834e-22⋅dt  + 7.2078645877627939543e-9⋅dt  + O⎝dt  ⎠
     (Coefficients of order 1e-15 or less may generally be ignored, as these are at roundoff error.)

# Step 3: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-RK_Butcher_Table_Validation.pdf](Tutorial-RK_Butcher_Table_Validation.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-RK_Butcher_Table_Validation.ipynb
!pdflatex -interaction=batchmode Tutorial-RK_Butcher_Table_Validation.tex
!pdflatex -interaction=batchmode Tutorial-RK_Butcher_Table_Validation.tex
!pdflatex -interaction=batchmode Tutorial-RK_Butcher_Table_Validation.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


# Voigt-Hjerting Function

Figure 2.5 from Chapter 2 of *Interstellar and Intergalactic Medium* by Ryden & Pogge, 2021,
Cambridge University Press.

Plot of the Voigt-Hjerting function $H(a,u)$ with a=0.0 to 3.0.  We use a numerical implementation of
the Voigt function as the real part of the [Faddeeva function](https://en.wikipedia.org/wiki/Faddeeva_function)
$w(z)$ in the SciPy special functions package.

```python
%matplotlib inline

import math
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator, LogLocator, NullFormatter

from scipy import special

import warnings
warnings.filterwarnings('ignore',category=UserWarning, append=True)
```

## Standard Plot Format

Setup the standard plotting format and make the plot. Fonts and resolution adopted follow CUP style.

```python
figName = 'Fig2_5'

# graphic aspect ratio = width/height

aspect = 4.0/3.0 # 4:3

# Text width in inches - don't change, this is defined by the print layout

textWidth = 6.0 # inches

# output format and resolution

figFmt = 'png'
dpi = 600

# Graphic dimensions

plotWidth = dpi*textWidth
plotHeight = plotWidth/aspect
axisFontSize = 10
labelFontSize = 6
lwidth = 0.5
axisPad = 5
wInches = textWidth
hInches = wInches/aspect

# Plot filename

plotFile = f'{figName}.{figFmt}'

# LaTeX is used throughout for markup of symbols, Times-Roman serif font

plt.rc('text', usetex=True)
plt.rc('font', **{'family':'serif','serif':['Times-Roman'],'weight':'bold','size':'16'})

# Font and line weight defaults for axes

matplotlib.rc('axes',linewidth=lwidth)
matplotlib.rcParams.update({'font.size':axisFontSize})

# axis and label padding

plt.rcParams['xtick.major.pad'] = f'{axisPad}'
plt.rcParams['ytick.major.pad'] = f'{axisPad}'
plt.rcParams['axes.labelpad'] = f'{axisPad}'
```

## Voigt-Hjerting Function

The Voigt function is the real part of the Faddeeva function.  We're slightly finessing the normalization
here for illustration purposes.

For this code, we're using [wofz()](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.wofz.html#scipy.special.wofz), the python
implementation of the Faddeeva function in the [SciPy special functions package](https://docs.scipy.org/doc/scipy/reference/special.html).

For a detailed derivation, the Wikipedia articles are good:
 * [Voigt Profile](http://en.wikipedia.org/wiki/Voigt_profile)
 * [Faddeeva Function](http://en.wikipedia.org/wiki/Faddeeva_function)

The Voigt-Hjerting function $H(a,u)$ can be written in terms of the Faddeeva function $w(z)$ as
$ H(a,u)=\Re[w(u + ia)]$.

We define a python function H(a,u) to make use of wofz() more transparent in what follows.

```python
def H(a,u):
    return special.wofz(u + 1j*a).real
```

## Compute H(a,u)

We will plot $H(a,u)$ for a range of $a$
\begin{equation}
a \equiv \frac{\gamma_{u\ell} c}{4 \pi \nu_{u\ell} b}
\end{equation}
and $u$:
\begin{equation}
u \equiv \frac{c}{b} \left( 1 - \nu/\nu_{u\ell}\right) = \frac{v}{b}  .
\end{equation}

Evaluate for $a$ from 0.0 to 3.0 in roughly logarithmic steps: a={0.0, 0.1, 0.3, 1.0, 3.0}, skipping 0.03 as too close to 0 to readily distinguish, and $u$ from -5 to +5 in steps of 0.05

```python
# Range of u: -5 to +5 in steps of 0.05

uMin = -5.0
uMax = 5.0
uStep = 0.05
numPts = int((uMax-uMin)/uStep)+1
u = np.linspace(uMin,uMax,numPts)

yMin = -0.05
yMax = 1.10

# range of a in 6 steps from 0 to 3.0

aRange = [0.0,0.1,0.3,1.0,3.0]
Hau = {}
for a in aRange:
    Hau[a]=H(a,u)
```

### Make the Plot

All lines are solid black and labeled with the value of $a$.

```python
fig,ax = plt.subplots()

fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

ax.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
ax.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

plt.xlim(uMin,uMax)
ax.xaxis.set_major_locator(MultipleLocator(2))
ax.xaxis.set_minor_locator(MultipleLocator(0.5))
plt.xlabel(r'u',fontsize=axisFontSize)

plt.ylim(yMin,yMax)
ax.yaxis.set_major_locator(MultipleLocator(0.2))
ax.yaxis.set_minor_locator(MultipleLocator(0.05))
plt.ylabel(r'H(a,u)',fontsize=axisFontSize)

# plot the curves

for a in aRange:
    plt.plot(u,Hau[a],'-',color='black',lw=1,zorder=10)
    if a == 0.0:
        label = 'a=0'
    else:
        label = f'{a:.1f}'

    plt.text(0.0,np.max(Hau[a]),label,color='black',va='bottom',ha='center',fontsize=labelFontSize)

# plot and file

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```


**Exercise 6: Minimax Risk Optimization**

We aim to recognize two different kinds of network traffic:
normal ($\omega_1$) and intrusive ($\omega_2$).
We assume that each sample $x$ is represented with a single feature
(numerical) value, and  that the likelihood of class $k$
(with $k=1,2$) is given as:

$$
\begin{equation}
p(x | \omega_k) = \mathcal N_k(x) = \mathcal N(x | \mu_k, \sigma_k) =
\frac{1}{\sqrt{2 \pi} \sigma_k}
\exp{\left
[-\frac{1}{2} \left (\frac{x-\mu_k}{\sigma} \right )^2
\right ]}.
\end{equation}
$$

The parameters for the two classes are: $\mu_1=0$, $\mu_2=4$, and $\sigma_1=\sigma_2=1$.

We suppose that *the prior probabilities are not available* (e.g. their estimation is not reliable, or they are time-variant, etc), while the cost matrix (predicted vs true class labels) is:

$$
\begin{equation}
\Lambda =
\begin{bmatrix}
\lambda_{1,1} & \lambda_{1,2} \\
\lambda_{2,1} & \lambda_{2,2}
\end{bmatrix} =
\begin{bmatrix}
0 & 2 \\
1 & 0
\end{bmatrix}.
\end{equation}
$$

Find the decision regions (and the decision rule) using the minimax criterion (which minimizes the maximum worst-case risk).

**Solution.** The total risk is given as:
$R = \int_{\mathcal R_1} \left ( \lambda_{1,1} P_1 \mathcal N_1 + \lambda_{1,2} P_2 \mathcal N_2 \right) dx + \int_{\mathcal R_2}\left(  \lambda_{2,1} P_1 \mathcal N_1 + \lambda_{2,2} P_2 \mathcal N_2\right) dx$.

By using the given costs $\Lambda$ and noting that $P_1 = 1-P_2$, we can simplify it as:
$R= 2 (1-P_1)\int_{\mathcal R_1}  \mathcal N_2 dx + P_1 \int_{\mathcal R_2}  \mathcal N_1  dx$.

Reordering and making explicit that the decision regions are $\mathcal R_1 = (-\infty, a)$ and $\mathcal R_2 = [a, +\infty)$, we get:

$$
\begin{equation}
R(a, P_1)= 2\int_{-\infty}^a  \mathcal N_2 dx + P_1 \left [ \int_{a}^\infty  \mathcal N_1  dx - 2\int_{-\infty}^a  \mathcal N_2 dx\right ] \, ,
\end{equation}
$$

where we also made it explicit that $R$ depends on two variables, namely, the decision boundary $a$ and the prior probability $P_1$. The former captures the shape of the decision regions, while the latter characterizes the data distribution.

For convenience, since $\int_{a}^\infty  \mathcal N_1  dx = 1-\int_{-\infty}^a  \mathcal N_1  dx$, the aforementioned risk function can be also written as:
$R(a, P_1)= 2\int_{-\infty}^a  \mathcal N_2 dx + P_1 \left [ 1-\int_{-\infty}^a  \mathcal N_1  dx - 2\int_{-\infty}^a  \mathcal N_2 dx\right ]$.

The minimax-risk decision regions, that is, the value of $a$, can be computed then by imposing the derivative of this function w.r.t. $P_1$ equal to zero, which simply amounts to (numerically) solving the following equation:
$ 1-\int_{-\infty}^a  \mathcal N_1  dx - 2\int_{-\infty}^a  \mathcal N_2 dx = 0$.

```
# finding the optimal value "a*"
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt

def grad_risk_p1(a):
  '''This is the derivative of the risk taken w.r.t. P1.'''
  return 1-norm.cdf(a,loc=mu1,scale=sigma1)-2*norm.cdf(a,loc=mu2,scale=sigma2)

mu1=0
mu2=4
sigma1=1
sigma2=1

a=np.arange(-5, 5, step=0.001)
grad_risk = grad_risk_p1(a)
plt.plot(a, grad_risk)
plt.plot(a, a*0, 'k')
plt.show()

#The optimal a* value lies at the intersection
a_star = max(a[grad_risk >= 0])
print('a*: ', np.round(a_star, decimals=4))
```

Thus, the decision regions are: $\mathcal R_1 = (-\infty, a^\star)$ and $\mathcal R_2 = [a^\star, +\infty)$, which means that we decide for $\omega_1$ if $x < a^\star$ and for $\omega_2$ if $x \geq a^\star$.

**Extra on minimax risk optimization.** Now, one may wonder how to compute the optimal value $P^\star_1$ that would allow us to compute the corresponding minimax threshold $\theta^\star$ on the likelihood ratio.

Recall indeed that the minimum-risk decision rule is:
$l(x) = \frac{p(x|\omega_1)}{p(x|\omega_2)} \gtrless\theta = \frac{\lambda_{12}-\lambda_{22}}{\lambda_{21}-\lambda_{11}}\frac{P_2}{P_1}$.

Thus, using the values in our exercise, the optimal minimax threshold would be given as $\theta^\star = 2\frac{1-P^\star_1}{P^\star_1}$. However, to compute such value we will need to find $P^\star_1$ corresponding to the optimal minimax solution.

Before doing that, we would like to clarify the minimax problem formulation from the mathematical perspective, using the given exercise as a specific instance of its more general formulation.
In general, the goal of minimax risk optimization is to find the decision regions (parametrized by $a$ in our case) that minimize the worst-case risk (maximized w.r.t. $P_1$). This is captured by the following optimization problem:

$$
\begin{equation}
\min_a \max_{P_1} R(a,P_1) = 2\int_{-\infty}^a  \mathcal N_2 dx + P_1 \left [ \int_{a}^\infty  \mathcal N_1  dx - 2\int_{-\infty}^a  \mathcal N_2 dx\right ] \, ,
\end{equation}
$$

where we also reported the explicit risk computation for our exercise.
The risk $R$ is a function of two variables. So we can plot it in the two-dimensional space spanned by $a$ and $P_1$ using a colormap to denote the values of $R$ (along the third axis).

```
min_a = -5
max_a = 5
min_p1 = 0
max_p1 = 1
res = 0.1

def R(p1, a, mu1, sigma1, mu2, sigma2):
    # This function computes the risk
    n1 = norm.cdf(a,loc=mu1,scale=sigma1)
    n2 = norm.cdf(a,loc=mu2,scale=sigma2)
    return 2 * n2 + p1 * (1-n1-2*n2)

def plot_fun(fun, only_at_zero=False, min_x=-5, max_x=+5, min_y=-5, max_y=+5,
             resolution=0.5, **kwargs):
    xx1, xx2 = np.meshgrid(np.arange(min_x, max_x, resolution),
                           np.arange(min_y, max_y, resolution))
    x = np.array([xx1.ravel(), xx2.ravel()]).T
    z = np.zeros(shape=(x.shape[0],))
    for i in range(x.shape[0]):
        z[i] = fun(x[i, 0], x[i, 1], **kwargs)
    z = z.reshape(xx1.shape)
    if not only_at_zero:
        plt.contourf(xx1, xx2, z, 30, cmap='Blues')
    else:
        plt.contour(xx1, xx2, z, [0], colors=['k'])
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    return

def find_min_R_a(min_p1, max_p1, min_a, max_a, res_p1, res_a, **kwargs):
    """
    For each p1, compute a* that gives the minimum value of R(a,p1).
    Then return the value of a* and R(a*, p1)
    """
    p1_vals = np.arange(min_p1, max_p1+res_p1, res_p1)
    a_vals = np.arange(min_a, max_a+res_a, res_a)
    min_a_vals = np.zeros((p1_vals.size,), dtype=float)
    min_r_vals = np.zeros((p1_vals.size,), dtype=float)

    for i, p1 in enumerate(p1_vals):
        r_vals = np.arange(a_vals.size, dtype=float)
        for j, a in enumerate(a_vals):
            r_vals[j] = R(p1, a, **kwargs)
        min_r_vals[i] = r_vals.min()
        min_a_vals[i] = a_vals[r_vals.argmin()]
    return p1_vals, a_vals, min_a_vals, min_r_vals

p1_vals, a_vals, min_a_vals, min_r_vals = find_min_R_a(
    min_p1, max_p1, min_a, max_a, res_p1=0.01, res_a=0.01,
    mu1=mu1, sigma1=sigma1, mu2=mu2, sigma2=sigma2)

fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 12
fig_size[1] = 5

plt.subplot(1,2,1)
plot_fun(R, False, min_p1, max_p1+res, min_a, max_a+res,
         res, mu1=mu1, sigma1=sigma1, mu2=mu2, sigma2=sigma2)
plt.colorbar()
plt.grid()
plt.plot(p1_vals, min_a_vals, c='k')
plt.plot(p1_vals, [a_star] * p1_vals.size, c='r')
plt.xlabel('p1', fontsize=12)
plt.ylabel('a', fontsize=12)
plt.title("R(a,p1)")

plt.subplot(1,2,2)

# value of the minimax optimal risk:
R_mm = 2 * norm.cdf(a_star,mu2,sigma2)
print('Minimax risk: ', R_mm)

plt.plot(p1_vals, min_r_vals, c='k')
plt.plot(p1_vals, [R_mm] * p1_vals.size, c='r')
plt.xlabel('p1', fontsize=12)
plt.title(r"$min_a R(a,p1)$")

plt.show()

```

As one can see from the leftmost plot, if we fix $a$, then the risk varies linearly as $P_1$ changes (the function varies linearly if we consider lines parallel to the x-axis), except when $a=a^\star=1.853$ (red line). For this value, in fact, the risk remains constant and equal to:

$
\begin{equation}
R_{\rm mm} = R(a^\star, P_1) = 2\int_{-\infty}^{a^\star}  \mathcal N_2 dx + P_1 \underbrace{\left [ \int_{a^\star}^\infty  \mathcal N_1  dx - 2\int_{-\infty}^{a^\star}  \mathcal N_2 dx\right ]}_{=0} = 2\int_{-\infty}^a  \mathcal N_2 dx \, .
\end{equation}
$

This is the minimax optimal risk.

The black line in the rightmost plot shows the values of $\min_a R(a, P^\prime_1)$, i.e., the minimum risk for any fixed $P^\prime_1$ value. These are the same values on the black line extrapolated from the leftmost plot. The red line shows instead the minimax risk $R_{\rm mm}$ computed at $a^\star$ and $P_1^\star$, which basically bounds the worst possible value found along the black line.

The goal now is to find not only the value of $a^\star$ but also the optimal value of $P_1$, i.e., $P_1^\star$, corresponding to the saddle point of the given problem.

In our example, they can be find analytically by setting the first derivatives of $R$ equal to zero (we did this only w.r.t. $P_1$ when solving the exercise to find $a^\star$ - but we need a second equation to find $P^\star_1$).

The derivatives are given as:

$$
\begin{eqnarray}
\frac{\partial R}{\partial P_1} &=& 1-\int_{-\infty}^a  \mathcal N_1(x)  dx - 2\int_{-\infty}^a  \mathcal N_2(x) dx = 0 \, , \\
\frac{\partial R}{\partial a} &=& 2 \mathcal N_2(a) - P_1 \left (\mathcal N_1(a) +2\mathcal N_2(a) \right) = 0 \, ,
\end{eqnarray}
$$

where we used the fact that $\frac{\partial }{\partial x} \int_{-\infty}^{x} f(v) dv = f(x)$ to compute the derivative in the second equation.

We know from the first equation that $a^\star \approxeq 1.853$ (via numerical solution).

By replacing this value in the second equation, we can find $P_1^\star = \frac{2 \mathcal N_2(a^\star)}{\mathcal N_1(a^\star) +2\mathcal N_2(a^\star)}\approxeq 0.526$, and the optimal threshold for the likelihood ratio $\theta^\star \approxeq 1.80$.

By imposing this threshold on the likelihood ratio, regardless of the priors, we will always observe a constant value of the risk (i.e., the worst-case one we computed before, $R_{\rm mm} \approxeq 0.032$).

```
n1 = norm.pdf(a_star, mu1, sigma1)
n2 = norm.pdf(a_star, mu2, sigma2)

p1_star = 2.0 * n2 / (n1+2.0*n2)
print("P_1*: ", p1_star)

theta_star = 2.0*(1-p1_star)/p1_star
print("theta*: ", theta_star)
```

    P_1*:  0.5262626031420929
    theta*:  1.8003840441232952


# 2 DOF System

The data that describes our system and its loading:

```python
M = np.array(((2.0, 0.0), ( 0.0, 1.0)))
K = np.array(((3.0,-2.0), (-2.0, 2.0)))
p = np.array(( 0.0, 1.0)); w = 2.0

print_mat(M, pre='\\boldsymbol{M}=m\\,', fmt='%d')
print_mat(K, pre='\\boldsymbol{K}=k\\,', fmt='%d')
print_mat(p[:,None], pre=r'\boldsymbol{p}(t) = p_0\,', fmt='%d',
          post='\\sin(%d\\omega_0t)'%w, mt='B')
```

$$\boldsymbol{M}=m\,\begin{bmatrix}2&0\\0&1\end{bmatrix}$$

$$\boldsymbol{K}=k\,\begin{bmatrix}3&-2\\-2&2\end{bmatrix}$$

$$\boldsymbol{p}(t) = p_0\,\begin{Bmatrix}0\\1\end{Bmatrix}\sin(2\omega_0t)$$

## Computing the eigenvalues and the eigenvectors

Using the `eigh` library function, specialized for Hermitian (symmetric) matrices.

We compute also the _starred_ matrices.  Note that the eigenvectors returned by `eigh` are normalized with respect to `M`, i.e., $M_i=m$ for every $i$.

```python
evals, Psi = eigh(K, M)
Mstar = Psi.T@M@Psi
Kstar = Psi.T@K@Psi
pstar = Psi.T@p
```

The `@` operator stands, in this context, for matrix multiplication.

We print the computed values.

```python
print_mat(evals[None,:], mt='p', pre=r'\omega^2_i=\omega^2_o\,')
print_mat(Psi, pre=r'\boldsymbol{\Psi}=')
print_mat(Mstar, pre=r'\boldsymbol{M}^\star=m\,')
print_mat(Kstar, pre=r'\boldsymbol{K}^\star=k\,')
print_mat(pstar[:,None], pre=r'\boldsymbol{p}^\star=p_o\,', mt='B')
```

$$\omega^2_i=\omega^2_o\,\begin{pmatrix}0.313859&3.186141\end{pmatrix}$$

$$\boldsymbol{\Psi}=\begin{bmatrix}-0.541774&-0.454401\\-0.642621&0.766185\end{bmatrix}$$

$$\boldsymbol{M}^\star=m\,\begin{bmatrix}1.000000&-0.000000\\-0.000000&1.000000\end{bmatrix}$$

$$\boldsymbol{K}^\star=k\,\begin{bmatrix}0.313859&0.000000\\0.000000&3.186141\end{bmatrix}$$

$$\boldsymbol{p}^\star=p_o\,\begin{Bmatrix}-0.642621\\0.766185\end{Bmatrix}$$

## Modal Response

The modal equation of motion, divided by $m$
$$ \ddot q_i + \Lambda^2_i\omega_0^2 q_i = \frac{p_0}{k}\frac{k}{m}\,p^\star_i \sin2\omega_0t.$$

The particular integral $\xi_i(t)$ is given by

$$\xi_i = \Delta_{st} \frac{\omega_0^2}{(\Lambda^2_i-4)\omega_0^2}p^\star_i\sin2\omega_0t.$$

Eventually the modal response, computed in the hypotesis of initial rest conditions is

$$q_i = \frac{\Delta_{st}}{\Lambda^2_i-4}p^\star_i(\sin2\omega_0t-\beta\sin\Lambda_i\omega_0t).$$

```python
L  = np.sqrt(evals)
DAF = 1.0/(L**2-w**2)
beta = w/L
t = np.linspace(0,60,601)[:,None]
q = pstar*DAF*(np.sin(w*t)-beta*np.sin(L*t))
```

The definition of time vector is a bit complicated...

```python
curves = plt.plot(t,q)
plt.legend(curves,['q1', 'q2'])
plt.title('Modal Response')
plt.xlabel('$\omega_0t$')
plt.ylabel('$q_i/\Delta_{st}$');
```

```python
x = (Psi@q.T).T

curves = plt.plot(t, x)
plt.legend(curves,['x1', 'x2'])
plt.title('Structural Response')
plt.xlabel('$\omega_0t$')
plt.ylabel('$X_i/\Delta_{st}$');
```

The following code cell (that is executed _before_ any other code cell by the notebook) loads libraries (or functions from libraries) and determines the style of plots and of the notebook itself.  Besides the cell defines a function to format conveniently matrices and vectors.

```python
%matplotlib inline

import matplotlib.pyplot as plt
plt.style.use(['fivethirtyeight', '00_mplrc'])

import numpy as np ; from scipy.linalg import eigh
np.set_printoptions(suppress=False, linewidth=120)

from IPython.display import Latex

def print_mat(mat, pre='', post='', fmt='%.6f', mt='b'):
    display(Latex(
        r'$$' + pre + r'\begin{%smatrix}'%mt +
        r'\\'.join('&'.join(fmt%x for x in row) for row in mat) +
        r'\end{%smatrix}'%mt + post + r'$$'))
```

```python
import sympy as sy
sy.init_printing(use_latex=1)
o = sy.symbols('Omega')
display(o)
sM = sy.Matrix(((2,0,),(0,1)))
sK = sy.Matrix(((3, -2),(-2,2)))
KooM = sK - o*sM
iKooM = KooM.inv()
sp = sy.Matrix(((0,),(1,)))
a,b=(iKooM*sp)
a.expand().simplify(), b.expand().simplify()
```

```python
with plt.style.context('classic'):
    plot = sy.plot(a, b, (o, 0, 5), ylim=(-10,10),
                   show=False, adaptive=False, nb_of_points=601)
    plot[0].line_color = 'black'; plot[0].label = '$x_1$'
    plot[1].line_color = 'red'  ; plot[1].label = '$x_2$'
    plot.xlabel = '$\\beta^2$'; plot.ylabel = r'$x_{i,\mathrm{ss}}/\Delta_\mathrm{st}$'
    plot.legend = True
    plot.show()
```

```python
from IPython.display import HTML
HTML(open('00_custom.css').read())
```

<style>
@font-face {
    font-family: 'Charis SIL';
    src: url('fonts/CharisSILEur-R.woff') format('woff');
}
@font-face {
    font-family: 'Architects Daughter';
    font-style: normal;
    src: url(https://fonts.gstatic.com/s/architectsdaughter/v6/RXTgOOQ9AAtaVOHxx0IUBM3t7GjCYufj5TXV5VnA2p8.woff2) format('woff2');
    unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215, U+E0FF, U+EFFD, U+F000;
}
.prompt{display: None;}
div.cell{margin: auto;width:1000px;}

div #notebook { /* centre the content */
    background: #fff;
    margin: auto;
    padding: 0em;
    padding-top: 1em;
}
div #notebook_container {width: 100px!important;}
#notebook li { /* More space between bullet points */
    margin-top:0.2em;
}

div.text_cell_render{
    font-family: "Charis SIL", Cambria,  serif;
    line-height: 155%;
    font-size: 130%;
}
.CodeMirror{
    font-family: Consolas, monospace;
    font-size: 140%;
    background-color:#fafcff!important;
}

.output_area {
    font-family: Consolas,monospace;font-size: 120%;
    background-color:#fafcff!important;
    margin-top:0.8em;
}

div.output_latex{overflow:hidden}

h1 {
    font-family: 'Architects Daughter', serif;
    font-size:   48pt!important;
    text-align:  center;
    text-shadow: 4px 4px 4px #aaa;
    padding-bottom: 48pt;
}

.warning{color: rgb( 240, 20, 20 )}
</style>

```python

```


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

## Probabilities and Expectations ##

A function $f$ on the plane is called a *joint density* if:
- $f(x, y) \ge 0$ for all $x$, $y$
- $\int_x \int_y f(x, y)dydx = 1$

If you think of $f$ as a surface, then the first condition says that the surface is on or above the plane. The second condition says that the total volume under the surface is 1.

Think of probabilities as volumes under the surface, and define $f$ to be the *joint density of random variables $X$ and $Y$* if

$$
P((X, Y) \in A) ~ = ~ \mathop{\int \int}_A f(x,y)dydx ~~~~~ \text{for all } A
$$

That is, the chance that the random point $(X, Y)$ falls in the region $A$ is the volume under the joint density surface over the region $A$.

This is a two-dimensional analog of the fact that probabilities involving a single continuous random variable can be thought of as areas under the density curve.

### Infinitesimals ###
Also analogous is the interpretation of the joint density as part of the calculation of the probability of an infinitesimal region.

The infinitesimal region is a tiny rectangle in the plane just around the point $(x, y)$. Its width is $dx$ and its length is $dy$. The corresponding volume is that of a rectangular box whose base is the tiny rectangle and whose height is $f(x, y)$.

Thus for all $x$ and $y$,
$$
P(X \in dx, Y \in dy) ~ \sim ~ f(x, y)dxdy
$$

and the joint density measures *probability per unit area*:
$$
f(x, y) ~ \sim ~ \frac{P(X \in dx, Y \in dy)}{dxdy}
$$

An example will help us visualize all this. Let $f$ be defined as follows:

$$
f(x, y) ~ = ~
\begin{cases}
120x(y-x)(1-y), ~~~ 0 < x < y < 1 \\
0 ~~~~~~~~ \text{otherwise}
\end{cases}
$$

For now, just assume that this is a joint density, that is, it integrates to 1. Let's first take a look at what the surface looks like.

### Plotting the Surface ###
To do this, we will use a 3-dimensional plotting routine. First, we define the joint density function. For use in our plotting routine, this function must take $x$ and $y$ as its inputs and return the value $f(x, y)$ as defined above.

```python
def joint(x,y):
    if y < x:
        return 0
    else:
        return 120 * x * (y-x) * (1-y)
```

Now we will call `Plot_3d` to plot the surface. The arguments are the limits on the $x$ and $y$ axes, the name of the function to be plotted, and two optional arguments `rstride` and `cstride` that determine how many grid lines to use. Larger numbers correspond to less frequent grid lines.

```python
Plot_3d(x_limits=(0,1), y_limits=(0,1), f=joint, cstride=4, rstride=4)
```

The surface has level 0 in the lower right hand triangle because it is not possible for the point $(X, Y)$ to be in that region.

The possible values of $(X, Y)$ are the blue region shown below. For calculations by hand, we will frequently draw just the possible values and not the surface.

```python
# HIDDEN

plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
xx = np.arange(0, 1.11, 0.1)
yy = np.ones(len(xx))
plt.fill_between(xx, xx, yy, alpha=0.3)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('Possible Values of $(X, Y)$');
```

### The Total Volume Under the Surface ###
The function $f$ looks like a bit of a mess but it is easy to see that it is non-negative. It's a good idea to check that the total probability under the surface is equal to 1.

To set up the double integral over the entire region of possible values, notice that for each fixed value of $y$, the value of $x$ goes from 0 to $y$. The values of $y$ go from 0 to 1.

We will first fix $y$ and integrate with respect to $x$. Then we will integrate with respect to $y$. Thus the total integral is

\begin{align*}
& ~~ \int_0^1 \int_0^y 120x(y-x)(1-y)dxdy \\
&= ~ 120 \int_0^1 (1-y) \Big{(}\int_0^y (xy - x^2)dx\Big{)}dy \\
&= ~ 20 \int_0^1 (1-y)y^3 dy ~ = ~  20 \big{(} \frac{1}{4} - \frac{1}{5}\big{)} ~ = ~  1
\end{align*}

#### By `SymPy` ####
To use `SymPy` we must first declare the two variables to have values in the unit interval, and assign the function to the name `f`. This specification doesn't say that $x < y$ but we will enforce that condition when we integrate.

```python
declare('x', interval=(0, 1))
declare('y', interval=(0, 1))
f = 120*x*(y-x)*(1-y)
```

The double integral requires a call to `Integral` that specifies the inner integral first and then the outer. The call says:
- The function being integrated is $f$.
- The inner integral is over the variable $x$ which goes from $0$ to y.
- The outer integral is over the variable $y$ which goes from 0 to 1.

```python
Integral(f, (x, 0, y), (y, 0, 1))
```

To evaluate the integral, use `doit()`:

```python
Integral(f, (x, 0, y), (y, 0, 1)).doit()
```

### Probabilities as Volumes ###
Probabilities are volumes under the joint density surface. In other words, they are double integrals of the function $f$. For each probability, we have to first identify the region of integration, which we will do by geometry and by inspecting the event. Once we have set up the integral, we have to calculate its value, which we will do by calculus or `SymPy`.

### Example 1 ###
Suppose you want to find $P(Y > 4X)$. The event is the blue region in the graph below.

```python
# HIDDEN
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
xx = np.arange(0, 0.251, 0.05)
yy = np.ones(len(xx))
plt.fill_between(xx, 4*xx, yy, alpha=0.3)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$Y > 4X$');
```

To find the region of integration, notice that for fixed $y$, the value of $x$ goes from 0 to $y/4$. So

\begin{align*}
P(Y > 4X) ~ &= ~ \int_0^1 \int_0^{y/4} 120x(y-x)(1-y)dxdy \\
&= ~ 120 \int_0^1 (1-y) \Big{(} \int_0^{y/4} (xy - x^2)dx \Big{)}dy \\
&= ~ 120\Big{(} \frac{1}{32} - \frac{1}{192} \Big{)} \int_0^1 (1-y) y^3dy \\
&= ~ 120\Big{(} \frac{1}{32} - \frac{1}{192}\Big{)} \cdot \frac{1}{20} ~ = ~ 0.15625
\end{align*}

#### By `SymPy` ####

```python
Integral(f, (x, 0, y/4), (y, 0, 1))
```

```python
Integral(f, (x, 0, y/4), (y, 0, 1)).doit()
```

```python
5/32
```

### Example 2 ###
Suppose you want to find $P(X > 0.25, Y > 0.5)$. The event is the colored region below.

```python
# HIDDEN
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
xx = np.arange(0.25, .52, 0.05)
yy1 = 0.5*np.ones(len(xx))
yy2 = np.ones(len(xx))
plt.fill_between(xx, yy1, yy2, alpha=0.3)
xx = np.arange(0.5, 1.1, 0.1)
yy1 = 0.5*np.ones(len(xx))
yy2 = np.ones(len(xx))
plt.fill_between(xx, xx, yy2, alpha=0.3)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$X > 0.25, Y > 0.5$');
```

Now $P(X > 0.25, Y > 0.5)$ is the integral of the joint density function over this region. Notice that for each fixed value of $y > 0.5$, the value of $x$ in this event goes from $0.25$ to $y$. So let's integrate $x$ first and then $y$, as we did in the previous calculations.

$$
P(X > 0.25, Y > 0.5) ~ = ~ \int_{0.5}^1 \int_{0.25}^y 120x(y-x)(1-y)dxdy
$$

You can crank that out by hand or use `SymPy`.

```python
Integral(f, (x, 0.25, y), (y, 0.5, 1))
```

```python
Integral(f, (x, 0.25, y), (y, 0.5, 1)).doit()
```

### Expectation ###
Let $g$ be a function on the plane. Then
$$
E(g(X, Y)) ~ = ~ \int_y \int_x g(x, y)f(x, y)dxdy
$$
provided the integral exists, in which case it can be carried out in either order: $x$ first and then $y$, or the other way around.

This is an application of the non-linear function rule for expectation, applied to two random variables with a joint density.

As an example, let's find $E(\frac{Y}{X})$ for $X$ and $Y$ with the joint density $f$ used in the examples above.

Here $g(x, y) = \frac{y}{x}$, and

\begin{align*}
E\big{(}\frac{Y}{X}\big{)} ~ &= ~ \int_y \int_x g(x, y)f(x, y)dxdy \\ \\
&= ~ \int_0^1 \int_0^y \frac{y}{x} 120x(y-x)(1-y)dx dy \\ \\
&= ~ 120 \int_0^1 y(1-y) \Big{(}\int_0^y (y-x)dx \Big{)}dy \\
&= ~ 60 \int_0^1 y^3(1-y) dy ~ = ~ 60 \cdot \frac{1}{20} ~ = ~ 3
\end{align*}

#### By `SymPy` ####
Remember that `x` and `y` have already been defined as symbolic variables with values in the unit interval.

```python
ev_y_over_x = Integral((y/x)*f, (x, 0, y), (y, 0, 1))
ev_y_over_x
```

```python
ev_y_over_x.doit()
```

```python

```