```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

## Numerical Integration

Numerical integration methods seek an approximate solution to a definite integral
$$\int_a^b f(x)\,  dx \, .$$

```python
# define function
def f(x):
    return 1/(1+np.power(x,2))

# domain bounds
a = 0
b = 3

# plot
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

x = np.linspace(a-0.1*(b-a), b+0.1*(b-a), 100)
y = f(x)
ax.plot(x, y,'-', label='$f(x)$')

X = np.linspace(a,b,100)
Y = f(X)
ax.fill_between(X,Y, color='r', alpha=0.2, label="$\int_a^b f(x)\,  dx$")

ax.legend(prop={'size':15})
plt.show()

```

In above example, we have chosen
$$f(x)=\frac{1}{1+x^2} \; .$$
The integral of this function is known:
$$\int \frac{1}{1+x^2} \, dx= \tan^{-1}(x) + C =\arctan(x) + C$$
So we can solve the integral analytically:
$$\int_a^b \frac{1}{1+x^2} \, dx = \arctan(b) - \arctan(a)$$

```python
# plot arctan(x)
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

y = np.arctan(x)
ax.plot(x, y,'-', label='arctan(x)')
ax.axvline(x=a, color='green', linestyle=':')
ax.axvline(x=b, color='red', linestyle=':')
ax.axhline(y=np.arctan(a), color='green', linestyle='--', label='arctan(x=a)')
ax.axhline(y=np.arctan(b), color='red', linestyle='--',  label='arctan(x=b)')

ax.legend(prop={'size':15})
plt.show()
```

However, not all functions can be integrated analytically, and even if a closed formulation exists, it may be easier to compute a numerical approximation rather than the antiderivative. Also, the integrand $f(x)$ may only be known at certain points.

### Riemann Sums

The [Riemann Integral](https://en.wikipedia.org/wiki/Riemann_integral) is a natural starting point for considering approximation methods for integrals.

Let's assume that $f(x)$ is a bounded function defined on the interval $[a, b]$ which is divided into $N$ subintervals of length $\Delta x = \left| a-b\right|/N$ defining a partition $\{x_0, x_1, \ldots x_{N}\}$ where $a=x_0 < x_1 < x_2 < \ldots <x_{N-1} < x_{N} =b$.

In each of the $N$ subintervals $[x_i, x_{i+1}]\; i=0, 1, \ldots, N-1$, the function $f(x)$ will have a maximum value $f_{\text{max, i}}$ and a minimum value $f_{\text{min, i}}$.
Defining $\Delta_i=x_{i+1}-x_i$, we can compute an upper and lower limit for the "area under the curve" $\mathcal{I}$ by summing the contribution of all subintervals:

\begin{align}
\text{lower bound:}& \qquad \mathcal{s}_N  =  \sum_{i=0}^{N-1} f_{\text{min, i}}\, \Delta_i \leq \mathcal{I} \tag{1a}\\
\text{upper bound:}& \qquad \mathcal{S}_N =  \sum_{i=0}^{N-1} f_{\text{max, i}}\, \Delta_i \geq \mathcal{I}\tag{1b}\\
\end{align}

These are called the *lower* and *upper* [Riemann Sum](https://en.wikipedia.org/wiki/Riemann_sum#Trapezoidal_rule), respectively.
As $N$ approaches infinity, i.e. $\Delta_i\rightarrow 0$, $\mathcal{s}_N$ and $\mathcal{S}_N$ converge to the exact value $\mathcal{I}$.

#### Rectangular Method: Left & Right Sums

Definitions (1a), (1b) can be used to approximate an integral numerically.
Instead of using the minimum or maximum of the function in each subinterval, we can use the value of the function on either side of each subinterval:

\begin{align}
\text{left sum:}& \qquad \mathcal{S}_{L, N}  =  \sum_{i=0}^{N-1} f(x_i)\, \Delta_i \approx \mathcal{I} \tag{2a}\\
\text{right sum:}& \qquad \mathcal{S}_{R, N} =  \sum_{i=0}^{N-1} f(x_{i+1})\, \Delta_i \approx \mathcal{I}\tag{2b}\\
\end{align}

These methods are examples of the **rectangular rule** because the area corresponding to each subinterval is approximated by a rectangle with dimensions length of the subinterval $\Delta_i$ and the function value at one of the endpoints of the subinterval.

The **left rule** will overestimate the actual value of the integral $\mathcal{I}$ in a region where f(x) decreases monotonically, and will overestimate its value where f(x) is increase monotonically. The opposite holds for the **right rule**.

#### Rectangular Method: Midpoint Rule

Another variation of the rectangular method is the **midpoint rule** , where the function value at the midpoint (instead of one of the endpoints) is used to compute the  rectangular area corresponding to each subinterval:

\begin{align}
\text{midpoint sum:}& \qquad \mathcal{S}_{M, N}  =  \sum_{i=0}^{N-1} f\left(\frac{x_i + x_{i+1}}{2}\right)\, \Delta_i \approx \mathcal{I} \tag{3}\\
\end{align}

As the comparison below shows, the midpoint rule provides a more accurate approximation of the integral than the rectangular method.

#### Comparison: Left, Right, Midpoint Rule

```python
# define function
def f(x):
    return 1/(1+np.power(x,2))

# domain bounds
a = 0
b = 3

# actual function, evaluated on finer grid to be represented smoothly
X = np.linspace(a-0.1*(b-a), b+0.1*(b-a), 100)
Y = f(X)

# partition for numeric approximation
N = 10
x = np.linspace(a, b, N+1)
y = f(x)
delta_x = (b-a)/(N)

# evaluate f(x) at left, right , center points of each interval
x_left = x[:-1]                             # left endpoints
y_left = y[:-1]
x_right = x[1:]                             # right endpoints
y_right = y[1:]
x_mid = (x[1:] + x[:-1])/2                  # mid points
y_mid = f(x_mid)

# compute approximation:
left_sum  = np.sum(y_left*delta_x)
midpt_sum = np.sum(y_mid*delta_x)
right_sum = np.sum(y_right*delta_x)
integral  = np.arctan(b) - np.arctan(a)

# function for creating textbox string
def make_txtbox_string(value_estimated, value_actual):
    textstr = '\n'.join((
    r'$\mathcal{S}=%-+.6f$' % (value_estimated, ),
    r'$\mathcal{I}=%-+.6f$' % (value_actual, ),
    r'$\mathcal{I}-\mathcal{S}=%+-.6f$' % (value_actual-value_estimated, )))
    return textstr

# plot
fig, axes= plt.subplots(1, 3, sharey=True, figsize=plt.figaspect(0.3))

# evaluation at left point
axes[0].plot(X,Y,'k')                            # function
axes[0].plot(x_left,y_left,'bo')   # numeric approximation (left value)
axes[0].bar(x_left,y_left,width=delta_x,
       alpha=0.2,align='edge',edgecolor='b')
axes[0].axvline(x=0,color='k', linestyle=':')
axes[0].set_title('Left Sum (N = %i)'%(N))
textstr_left = make_txtbox_string(left_sum, integral)
axes[0].text(3, 0.95, textstr_left,
             verticalalignment='top', horizontalalignment='right')
# evaluation at mid point
axes[1].plot(X,Y,'k')                            # function
axes[1].plot(x_mid,y_mid,'go')   # numeric approximation (left value)
axes[1].bar(x_mid,y_mid,width=delta_x,
       alpha=0.2,edgecolor='g')
axes[1].axvline(x=0,color='k', linestyle=':')
axes[1].set_title('Midpoint Sum (N = %i)'%(N))
textstr_midpt = make_txtbox_string(midpt_sum, integral)
axes[1].text(3, 0.95, textstr_midpt,
             verticalalignment='top', horizontalalignment='right')
# evaluation at right point
axes[2].plot(X,Y,'k')                            # function
axes[2].plot(x_right,y_right,'ro')   # numeric approximation (left value)
axes[2].bar(x_right,y_right,width=-delta_x,
       alpha=0.2,align='edge',edgecolor='r')
axes[2].axvline(x=0,color='k', linestyle=':')
axes[2].set_title('Right Sum (N = %i)'%(N))
textstr_right = make_txtbox_string(right_sum, integral)
axes[2].text(3, 0.95, textstr_right,
             verticalalignment='top', horizontalalignment='right')

plt.show()

```

##### Riemann Sums computed over Analytic Functions

---
**Exercise (1):**

Define a function that computes *left*, *right* and *midpoint* rules. The function should take 5 parameters:
-  *f*:    one-variable vectorized function
- *a*, *b*: scalar, lower and upper bound of integration interval $[a, b]$
- *N*:    scalar, number of subintervals for partitioning $[a, b]$
- *method*: string, type of sum, either 'left', 'right', 'midpoint'

---

```python
def riemann_sum(f, a, b, N, method='left'):
  """
  Returns riemann sum of function `f` over interval [a, b].

  Args:
    - f:    one-variable function
    - a, b: scalar, lower and upper bound of integration interval
    - N:    scalar, number of subintervals for partitioning [a, b]
    - method: string, type of sum, either 'left', 'right', 'midpoint'

  Returns:
    - integral approximation based on left, right, or midpoint method.
  """
  delta_x = np.abs(b-a)/N
  x = np.linspace(a, b, N+1)

  x_left = x[:-1]
  x_right = x[1:]
  if method == 'left':
    return np.sum(f(x_left)*delta_x)
  elif method == 'right':
    return np.sum(f(x_right)*delta_x)
  elif method == 'midpoint':
    x_midpt = (x_right + x_left)/2
    return np.sum(f(x_midpt)*delta_x)
  else:
    print("Method '%s' is undefined."%method)

```

##### Convergence Behavior

---

**Exercise (2):**

Use the function defined in the previous exercise to compare the convergence behaviour for *left*, *right* and *midpoint* quadratures.

- Choose a function $f(x)$ that can be integrated analytically. Compute the definite integral $\mathcal{I}$ of that function on the interval $[a,b]$.
- Approximate the definite integral by left, right and midpoint sum using some number $N$ of partitioning intervals . Compute the approximation error.
- Repeat this procedure for different partitionings. Start with $N=1$ and double the number of intervals until $\Delta_i \approx 10^{-5}$.
- Plot the approximation error in function of $\Delta_i$, use logarithmic axes.
- Compute the convergence order using formula (7) in from the [numerical differentiation notebook](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/03_IntroCompMethods_NumericalDifferentiation.ipynb).
---

```python
# define function
# we use again f(x)=1/(1+x2) with antiderivative F(x)=arctan(x)
def f(x):
  return 1/(1+np.power(x,2))

# integration interval
a = 0
b = 3

# actual value of integral
integral = np.arctan(b) - np.arctan(a)

# compute error for different partitions
# we start with N=1 and double the number of intervals, i.e.
# half the distance between partition points in each iteration
dfs = {}
for method in ['left', 'right', 'midpoint']:
  df = pd.DataFrame()
  for i in range(0,20):
      N = 1 * (2**i)
      delta_x = (b-a)/N
      integral_approx = riemann_sum(f, a, b, N, method)
      # keep record of parameters
      df.loc[i, 'method'] = method
      df.loc[i, 'N'] = N
      df.loc[i, 'delta_x'] = delta_x
      df.loc[i, 'integral_approx'] = integral_approx
  df['error'] = np.abs(integral - df['integral_approx'])
  dfs[method] = df

```

```python
# plot errors

fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

for method in ['left', 'right', 'midpoint']:
  df = dfs[method]
  ax.loglog(df.delta_x.values, df.error.values, marker='o',
            linestyle='-', label=method)

ax.set_xlabel('$\Delta_i$')
ax.set_ylabel("absolute error")
ax.set_title('Error in Integral Approximations using Rectangular Quadratures')
ax.legend()
plt.show()
```

```python
# we used this function previously in the numerical differentiation module

def compute_convergence_order(eps):
  """
  Computes ratio of errors steps: ratio(n) = error(n-1)/error(n)
  """
  eps_shifted = np.roll(eps, shift=1)
  eps_shifted[0] = np.nan
  eps_ratio = eps_shifted / eps
  convergence_order = np.log2(eps_ratio)
  return convergence_order
```

```python
# convergence order
for method in ['left', 'right', 'midpoint']:
  df = dfs[method]
  print("Convergence order of method '%s': "%(method),
       compute_convergence_order(df.error.values))

```

#### Riemann Sums computed over Grids

Until now we have a assumed that an analytic form of the integrand $f(x)$ is known that can be evaluated at arbitrary points $x_i$.
This is not always the case. In fact, most often only the values $f(x_i)$ and positions $x_i$ at which  $f(x)$ has been sampled will be known, but not the function $f(x)$ itself.

---
**Exercise (3):**

1. Rewrite the numerical integration function from the previous exercise to take as input parameters
   - an array of sampled function values $f(x_i)$
   - an array of sampling positions $x_i$

  Only consider *left* and *right rules*.
 Assume that the integral is to be performed over the entire array of function values, i.e. assume that the integration bounds correspond to $x_0$ and $x_N$.

2. Why is implementing the midpoint rule not straightforward?

3. If we assume that the function $f(x)$ has been sampled on a *regular grid* with equal spacings between sampling points $\Delta_i=x_{i+1}-x_i$, then *left* and *right* sums can be computed from  $\Delta_i$ without knowing each $x_i$ in the partition.

  How could you extend your function to accept as input either an array of sampling positions $x_i$, or  the spacing between sampling points $\Delta_i$.

---

```python
def riemann_sum_grid(data, x, method='left'):
  """
  Returns riemann sum of data/function values sampled at positions x_i.

  Args:
    - data: an array of data points / function values
    - x:    can be a scalar, indicating the spacing between subsequent
            observation/evaluation points, or
            an array of same length as data, indicating the observation points
    - method: string, type of sum, either 'left', or 'right'
  """
  if isinstance(x, np.ndarray):
    delta_x = np.roll(x, -1) - x       # dx = x_i+1 - x_i
    delta_x = delta_x[:-1]             # for i = 0 ... N-1
  elif (isinstance(x, float) or isinstance(x, int)):
    delta_x = x

  if method == 'left':
    data_sel = data[:-1]
  elif method == 'right':
    data_sel = data[1:]

  riemann_sum = np.sum(delta_x * data_sel)
  return riemann_sum
```

```python
# define function to be integrated
# we use again f(x)=1/(1+x2) with antiderivative F(x)=arctan(x)
def f(x):
  return 1/(1+np.power(x,2))

# integration interval
a = 0
b = 3

# actual value of integral
integral = np.arctan(b) - np.arctan(a)

# compute error for different partitions
# we start with N=1 and double the number of intervals, i.e.
# half the distance between partition points in each iteration
dfs = {}
for method in ['left', 'right']:
  df = pd.DataFrame()
  for i in range(0,20):
      # partition for numeric approximation
      N = 1 * (2**i)
      x = np.linspace(a, b, N+1)
      y = f(x)
      delta_x = (b-a)/(N)
      integral_approx = riemann_sum_grid(y, x, method)
      # keep record of parameters
      df.loc[i, 'method'] = method
      df.loc[i, 'N'] = N
      df.loc[i, 'delta_x'] = delta_x
      df.loc[i, 'integral_approx'] = integral_approx
  df['error'] = np.abs(integral - df['integral_approx'])
  dfs[method] = df
```

```python
# plot errors

fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

for method in ['left', 'right']:
  df = dfs[method]
  ax.loglog(df.delta_x.values, df.error.values, marker='o',
            linestyle='-', label=method)

ax.set_xlabel('$\Delta_i$')
ax.set_ylabel("absolute error")
ax.set_title('Error in Integral Approximations using Rectangular Quadratures')
ax.legend()
plt.show()
```

```python
# convergence order
for method in ['left', 'right']:
  df = dfs[method]
  print("Convergence order of method '%s': "%(method),
       compute_convergence_order(df.error.values))
```

### Quadrature based on Interpolation

We have seen previously that the midpoint quadrature converges to the true value of the integral more quickly than the rectangular quadrature.
However, in situations when numeric integration is most useful,  the 'midpoint' value is not known nor can it be evaluated.

Since we do not know the functional form of $f(x)$ for $x\in[x_i, x_{i+1}]$, we can only approximate its values by *interpolation*.
Indeed, a whole class of quadrature rules is *based on interpolating functions*. Instead of computing the integral of $f(x)$, we may approximate the true value of the integral by integrating an interpolant of $f(x)$.

An important class of interpolants are polynomials:
Given the values $f(x_0), \ldots, f(x_N)$ on $N+1$ points, any function $f(x)$ can be interpolated by a polynomial $P_N(x)$ of degree $\leq N$.

We have already used this approach:
For the left, right and midpoint *rectangular quadrature* rules, we simply approximated the function $f(x)$ in the interval $x\in[x_i, x_{i+1}]$ by a constant, i.e. a polynomial of degree 0, corresponding to the value of the function at either $x_i$, $x_{i+1}$, or the midpoint between both.

#### Trapezoidal Rule

Instead of approximating the function's value with information from only a single point $x_i$, we could use information from two neighbouring points.
This would allow us to interpolate $f(x)$ by a polynomial of degree 1, i.e. a linear function that connects both endoints of the interval $[x_i, x_{i+1}]$.

What would such a linear interpolation look like?

For readibility we rename the boundaries of the integration (sub)interval to $a$ and $b$, and we seek a polynomial of degree 1, $P_1(x)$, that interpolates the function f(x) for $x\in [a, b]$.

Surely, for $x=a$ and $x=b$, the interpolating function should recover the actual function values, i.e. $P_1(a)=f(a)$ and $P_1(b)=f(b)$.
At all positions $x \in (a, b)$ for which no information about the value of $f(x)$ is available, we weight the contribution of $f(a)$ and $f(b)$ by the distance of $x$ from points $a$ and $b$ repectively:
$$f(x) \approx P_1(x) = f(a) \frac{b-x}{b-a} + f(b)\frac{x-a}{b-a} \qquad x\in[a,b] \tag{4}$$

Having found an interpolation of $f(x)$, we can now approximate the integral:

\begin{align}
\int_a^b f(x)\, dx \approx \int_a^b P_1(x)\, dx &= f(a) \int_a^b \frac{b-x}{b-a} \, dx+ f(b) \int_a^b \frac{x-a}{b-a}\, dx \\
                                                                                &=  f(a) \frac{b-a}{2} + f(b)\frac{b-a}{2} \\
                                                                                &= \frac{b-a}{2}\left(f(a) + f(b)\right) \tag{5}
\end{align}

Geometrically, this means that the 'area under the curve' in each subinterval $[a,b]=[x_i, x_{i+1}]$ is approximated, not by a rectangle as before, but by a trapezoid.
This quadrature is therefore called the **trapezoidal rule** or **trapezoidal quadrature**.

Applied to the partition $\{x_0, x_1, \ldots x_{N}\}$ from before:

\begin{align}
\text{trapezoid sum} \qquad \mathcal{S}_{T, N}  &=\sum_{i=0}^{N-1} \frac{x_{i+1} - x_i}{2}\, \left[ f(x_{i+1}) + f(x_{i}) \right] \tag{6a}\\
                                                                                &= \frac{x_{1} - x_0}{2}\, \left[ f(x_{1}) + f(x_{0}) \right]
                                                                                + \frac{x_{2} - x_1}{2}\, \left[ f(x_{2}) + f(x_{1}) \right]
                                                                                + \dots
                                                                                + \frac{x_{N-1} - x_{N-2}}{2}\, \left[ f(x_{N-1}) + f(x_{N-2}) \right]
                                                                                + \frac{x_{N} - x_{N-1}}{2}\, \left[ f(x_{N}) + f(x_{N-1}) \right]\\
                                                                                &= f(x_{0})\frac{\Delta x}{2} +  2 f(x_{1})\frac{\Delta x}{2} +
                                                                                2 f(x_{2})\frac{\Delta x}{2} +\dots + 2 f(x_{N-2})\frac{\Delta x}{2} +
                                                                                2 f(x_{N-1})\frac{\Delta x}{2} + f(x_{N})\frac{\Delta x}{2}\\
                                                                                &=  f(x_{0})\frac{\Delta x}{2}
                                                                                +  \sum_{i=1}^{N-1} f(x_i)\, \Delta x
                                                                                + f(x_{N})\frac{\Delta x}{2} \tag{6b}
\end{align}

---
**Exercise (4):**

1. Define a function that uses the *trapezoidal quadrature* to compute an integral approximation.
  The function should take 2 parameters:

   - *data*: an array of sampled function values $f(x_i)$
   - *x*: an array of sampling positions $x_i$

  Assume that the integral is to be performed over the entire array of function values, i.e. assume that the integration bounds correspond to $x_0$ and $x_N$.

2.  Extend your function to accept as input either an array of sampling positions $x_i$, or  the spacing between sampling points $\Delta_i$, i.e. to compute (6a) and (6b).

---

```python
def trapezoidal_sum_grid(data, x):
  """
  Returns riemann sum of data / function using trapezoidal quadrature.

  Args:
    - data: an array of data points / function values
    - x:    can be a scalar, indicating the spacing between subsequent
            observation/evaluation points, or
            an array of same length as data, indicating the observation points
    - method: string, type of sum, either 'left', or 'right'
  """
  if isinstance(x, np.ndarray):
    data_i_plus_iplus1 = data + np.roll(data,-1)  # f(x_i) + f(x_i+1)
    data_i_plus_iplus1 = data_i_plus_iplus1[:-1]  # for i = 0 ... N-1
    delta_x = np.roll(x, -1) - x                  # dx = x_i+1 - x_i
    delta_x = delta_x[:-1]                        # for i = 0 ... N-1
    return np.sum(0.5 * data_i_plus_iplus1 * delta_x)
  elif (isinstance(x, float) or isinstance(x, int)):
    delta_x = x
    weights = np.ones_like( data )
    weights[0] = 0.5
    weights[-1]= 0.5
    return np.sum(delta_x * weights * data)

```

```python
# define function
# we use again f(x)=1/(1+x2) with antiderivative F(x)=arctan(x)
def f(x):
  return 1/(1+np.power(x,2))

# integration interval
a = 0
b = 3

# actual value of integral
integral = np.arctan(b) - np.arctan(a)

# compute error for different partitions
# we start with N=1 and double the number of intervals, i.e.
# half the distance between partition points in each iteration
df = pd.DataFrame()
for i in range(0,20):
    # partition for numeric approximation
    N = 1 * (2**i)
    x = np.linspace(a, b, N+1)
    y = f(x)
    delta_x = (b-a)/(N)
    integral_approx = trapezoidal_sum_grid(y, x)
    #integral_approx = trapezoidal_sum_grid(y, delta_x)

    # keep record of parameters
    df.loc[i, 'N'] = N
    df.loc[i, 'delta_x'] = delta_x
    df.loc[i, 'integral_approx'] = integral_approx
df['error'] = np.abs(integral - df['integral_approx'])

```

```python
# plot errors

fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

ax.loglog(df.delta_x.values, df.error.values, marker='o',
          linestyle='-', label='trapezoidal quadrature')

ax.set_xlabel('$\Delta_i$')
ax.set_ylabel("absolute error")
ax.set_title('Error in Integral Approximations using Trapezoidal Quadrature')
ax.legend()
plt.show()
```

```python
# convergence order
print("Convergence order of trapezoidal quadrature: ",
     compute_convergence_order(df.error.values))
```

#### Higher Order Quadratures

The procedure we have used to derive the trapezoidal rule above also allows to derive higher order quadratures by first finding an interpolant that appoximates $f(x)$ on the interval $[a, b]$ and then integrating that interpolant in place of $f(x)$.
Standard methods exist for finding interpolants of a function, given its values at specific locations. The approach demonstrated for the derivation of the trapezoidal used a first degree [Lagrange Polynomial](https://en.wikipedia.org/wiki/Lagrange_polynomial).

## Useful Python / Numpy Functions

```python
# np.trapz implements the trapezoidal rule

import numpy as np
# we use again f(x)=1/(1+x2) with antiderivative F(x)=arctan(x)
def f(x):
  return 1/(1+np.power(x,2))

# integration interval
a = 0
b = 3

# partition
N = 100
delta_x = (b-a)/(N)
x = np.linspace(a, b, N+1)

# values
y = f(x)

# integration using x
integral_trapez_our = trapezoidal_sum_grid(y, x)
integral_trapez_np  = np.trapz(y, x=x)

print("our implementation (given array x): ", integral_trapez_our)
print("np.trapz implementation (given array x): ", integral_trapez_np)

# integration using delta_x
integral_trapez_our = trapezoidal_sum_grid(y, delta_x)
integral_trapez_np  = np.trapz(y, x=None, dx=delta_x)

print("our implementation (given delta x): ", integral_trapez_our)
print("np.trapz implementation (given delta x): ", integral_trapez_np)
```

```python
# scipy.integrate: trapz, simps
import scipy.integrate as integrate

import numpy as np
# we use again f(x)=1/(1+x2) with antiderivative F(x)=arctan(x)
def f(x):
  return 1/(1+np.power(x,2))

# integration interval
a = 0
b = 3

# partition
N = 100
delta_x = (b-a)/(N)
x = np.linspace(a, b, N+1)

# values
y = f(x)

# integration using x
integral_trapez_our = trapezoidal_sum_grid(y, x)
integral_trapz = integrate.trapz(y, x)
#integral_cumtrapz = integrate.cumtrapz(y, x)
integral_simps = integrate.simps(y, x)

print("our implementation: ", integral_trapez_our)
print("trapz   : ", integral_trapz)     # uses linear interpolant as discussed
#print("cumtrapz: ", integral_cumtrapz)
print("simps   : ", integral_simps)     # uses quadratic interpolant

# above functions operate on discrete observations
# scipy.integrate also provides functions for numeric integration provided a
# function object
```

## Exercises

- In [this](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/exercises/05_NumericalIntegration.ipynb) exercise you approximate an definite integral using the trapezoidal quadrature. (**Optional**)
- In [this](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/exercises/06_NumericalIntegration_ComputeErrorFunction.ipynb) exercise you compute the *error function* using numerical integration. (**Optional**)

###### About
This notebook is part of the *biosci670* course on *Mathematical Modeling and Methods for Biomedical Science*.
See https://github.com/cohmathonc/biosci670 for more information and material.


# Predefined Metrics in Symbolic Module

### Importing some of the predefined tensors. All the metrics are comprehensively listed in EinsteinPy documentation.

```python
import sympy
from sympy import simplify
from einsteinpy.symbolic import RicciScalar
from einsteinpy.symbolic.predefined import Schwarzschild, DeSitter, AntiDeSitter, Minkowski, find

sympy.init_printing()  # for pretty printing
```

### Printing the metrics for visualization
All the functions return instances of :py:class:`~einsteinpy.symbolic.metric.MetricTensor`

```python
sch = Schwarzschild()
sch.tensor()
```

$\displaystyle \left[\begin{matrix}1 - \frac{r_{s}}{r} & 0 & 0 & 0\\0 & - \frac{1}{c^{2} \left(1 - \frac{r_{s}}{r}\right)} & 0 & 0\\0 & 0 & - \frac{r^{2}}{c^{2}} & 0\\0 & 0 & 0 & - \frac{r^{2} \sin^{2}{\left(\theta \right)}}{c^{2}}\end{matrix}\right]$

```python
Minkowski(c=1).tensor()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & 1.0 & 0 & 0\\0 & 0 & 1.0 & 0\\0 & 0 & 0 & 1.0\end{matrix}\right]$

```python
DeSitter().tensor()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & e^{\frac{2 x}{\alpha}} & 0 & 0\\0 & 0 & e^{\frac{2 x}{\alpha}} & 0\\0 & 0 & 0 & e^{\frac{2 x}{\alpha}}\end{matrix}\right]$

```python
AntiDeSitter().tensor()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & \cos^{2}{\left(t \right)} & 0 & 0\\0 & 0 & \cos^{2}{\left(t \right)} \sinh^{2}{\left(\chi \right)} & 0\\0 & 0 & 0 & \sin^{2}{\left(\theta \right)} \cos^{2}{\left(t \right)} \sinh^{2}{\left(\chi \right)}\end{matrix}\right]$

### Calculating the scalar (Ricci) curavtures
They should be constant for De-Sitter and Anti-De-Sitter spacetimes.

```python
scalar_curvature_de_sitter = RicciScalar.from_metric(DeSitter())
scalar_curvature_anti_de_sitter = RicciScalar.from_metric(AntiDeSitter())
```

```python
scalar_curvature_de_sitter.expr
```

```python
scalar_curvature_anti_de_sitter.expr
```

On simplifying the expression we got above, we indeed obtain a constant

```python
simplify(scalar_curvature_anti_de_sitter.expr)
```

### Searching for a predefined metric
find function returns a list of available functions

```python
find("sitter")
```

    ['AntiDeSitter', 'AntiDeSitterStatic', 'DeSitter']

docs/source/examples/Symbolic Module - Predefined Metrics.ipynb


```python
import numpy as np
import scipy.stats as stats
import matplotlib as mpl
import matplotlib.pyplot as plt
```

### Probability density functions

The probability denisty function for a (continuous) random variable is a function whose value at a given point, tells us the relative likelihood that the random variable takes that value. Below, is the probability density function for a Gaussian distribution with mean 0 and standard deviation 1, the so called bell curve.

```python
fig, ax = plt.subplots(1,1, figsize=(4.5,4.0))

x = np.arange(-4,4,0.1)
ax.plot(x, stats.norm.pdf(x,0,1))

ax.set_ylim((0,0.45))
ax.set_title('Gaussian probability density function')
ax.set_xlabel('$x$')
ax.set_ylabel('Density');
```

This function achieves its maximum at $x=0$, the mean, and gets smaller as you move further from this point. In reality, the probability that the random variable takes a particular, exact value is 0. In order to get the probability that the random variable lies in a particular region, we compute the area under the curve in that region. From this, we know that the area under the whole curve must give probabilty 1 as the random variable must take some value!

```python
fig, axs = plt.subplots(1,2, figsize=(10.0,4.0))

x = np.arange(-4,4,0.1)
axs[0].plot(x, stats.norm.pdf(x,0,1))

region = np.arange(-1,1,0.1)
axs[0].fill_between(region, stats.norm.pdf(region,0,1), facecolor='black', alpha=0.25)
axs[0].text(0, 0.1, 'Probability\n0.68', ha='center')

axs[0].set_ylim((0,0.45))
axs[0].set_title('Gaussian probability density function')
axs[0].set_xlabel('$x$')
axs[0].set_ylabel('Density')

axs[1].plot(x, stats.norm.pdf(x,0,1))

region = np.arange(-2,2,0.1)
axs[1].fill_between(region, stats.norm.pdf(region,0,1), facecolor='black', alpha=0.25)
axs[1].text(0, 0.1, 'Probability\n0.95', ha='center')

axs[1].set_ylim((0,0.45))
axs[1].set_title('Gaussian probability density function')
axs[1].set_xlabel('$x$')
axs[1].set_ylabel('Density');
```

In this example, the probability that the random variable is between -1 and 1 is 0.68. This is the probability that the random variable is within one standard deviation from the mean; it would also be 0.68 regarding of the mean and standard deviation of the Gaussian distibution. Increase this to two standard deviations, then the probability that the random variable is between -2 and 2 is 0.95. For a Gaussian distribution with mean $\mu$ and standard deviation $\sigma$, we denote the probability denisty function as $f(x; \mu, \sigma^2)$ which has the following equation:
\begin{equation}
    f(x; \mu, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp  \left( -\frac{1}{2 \sigma^2} (x - \mu)^2 \right).
\end{equation}

Given the probability density function, you can compute the probability of any event for the random variable by finding the relevant area under the curve. This process can also be reversed. If you know the probabilities of all the events, you can construct the probability density curve.

### Mixture models

Mixture models are a way to construct random variables combining other distributions. Suppose that we have two distributions; the first a Gaussian distribution with mean 0 and standard deviation 1 (probability density function $f(x; 0, 1)$), the second a Gaussian distribution with mean 4 and standard deviation 1 (probability density function $f(x; 4, 1)$). we toss a fair coin and given the outcome we randomly sample a point from either the first or second distribution. This process itself describes a random process so it will have its own probability density function that we will compute now.

```python
fig, ax = plt.subplots(1,1, figsize=(4.5,4.0))

x = np.arange(-4,8,0.1)
ax.plot(x, 0.5*stats.norm.pdf(x,0,1) + 0.5*stats.norm.pdf(x,4,1))
ax.plot(x, 0.5*stats.norm.pdf(x,0,1), c='tab:blue', ls='--')
ax.plot(x, 0.5*stats.norm.pdf(x,4,1), c='tab:blue', ls='--')

ax.set_ylim((0,0.25))
ax.set_title('Gaussian probability density function')
ax.set_xlabel('$x$')
ax.set_ylabel('Density');
```

The two dashed curves are the probability density functions $f(x; 0, 1)$ and $f(x; 4, 1)$ each scaled by the 0.5, the probability that the coin selects that random variable. The probability density function of our final random variable is given by the solid line and is the sum of the weighted probability density functions,
\begin{equation}
    f_{\text{Mixture}}(x) = 0.5 f(x; 0, 1) + 0.5 f(x; 4, 1).
\end{equation}

Given an observation, say $x=3.0$, we can ask which of the two distributions the point came from, basically asking was the coin's outcome heads or tails? In this case, it seems much more likely that the observation was drawn from the right distribution than the left, although it's not impossible. We can answer this by seeing how much of the two probablity density functions contribute to the mixture density function;
\begin{equation}
    \text{Probability observation $x$ is from left distribution} = \frac{0.5 f(x; 0, 1)}{0.5 f(x; 0, 1) + 0.5 f(x; 4, 1)},
\end{equation}
and likewise for the other distribution,
\begin{equation}
    \text{Probability observation $x$ is from right distribution} = \frac{0.5 f(x; 4, 1)}{0.5 f(x; 0, 1) + 0.5 f(x; 4, 1)}.
\end{equation}

We construct a function that allow us to answer this question for any number of Gaussian distributions with differing means and standard deviations, and a biased coin to choose between these distributions.

```python
def mixture_probability(x, means, stdevs, pis):
    n = len(means)
    probs = np.zeros(n)
    for i in range(n):
        probs[i] = pis[i]*stats.norm.pdf(x,means[i],stdevs[i])

    return probs/np.sum(probs)
```

```python
x = 3.0
probs = mixture_probability(x, [0,4], [1,1], [0.5,0.5])

print('Probability observation', x, 'is from left distribution: ', np.round(probs[0], 3))
print('Probability observation', x, 'is from right distribution:', np.round(probs[1], 3))
```

    Probability observation 3.0 is from left distribution:  0.018
    Probability observation 3.0 is from right distribution: 0.982

From this we see that it is much more likely that the point came from the right distribution.

Let's finish up with a slightly more complicated example. This time we have three Gaussian distributions, the first has mean 3 and standard deviation 0.5, the second mean 0 and standard deviation 1.0, and the third mean 4 and standard deviation 1.5. This time we will assume that the coin randomly chooses between these distributions with probablity given by $\pi = (0.5,0.3,0.2)$.

```python
fig, ax = plt.subplots(1,1, figsize=(4.5,4.0))

x = np.arange(-6,8,0.1)
ax.plot(x, 0.5*stats.norm.pdf(x,0,1) + 0.3*stats.norm.pdf(x,4,1.5) + 0.2*stats.norm.pdf(x,-3,0.5))
ax.plot(x, 0.2*stats.norm.pdf(x,-3,0.5), c='tab:blue', ls='--')
ax.plot(x, 0.5*stats.norm.pdf(x,0,1), c='tab:blue', ls='--')
ax.plot(x, 0.3*stats.norm.pdf(x,4,1.5), c='tab:blue', ls='--')

ax.set_ylim((0,0.25))
ax.set_title('Gaussian probability density function')
ax.set_xlabel('$x$')
ax.set_ylabel('Density');
```

Like before we can ask what the probability a particular observation comes from each of the distributions using the same formula, this time using the different standard deviations and coin probabilities.

```python
x = -2.0
probs = mixture_probability(x, [-3,0,4], [0.5,1,1.5], [0.5,0.3,0.2])

print('Probability observation', x, 'is from left distribution  :', np.round(probs[0], 3))
print('Probability observation', x, 'is from middle distribution:', np.round(probs[1], 3))
print('Probability observation', x, 'is from right distribution :', np.round(probs[2], 3))
```

    Probability observation -2.0 is from left distribution  : 0.769
    Probability observation -2.0 is from middle distribution: 0.231
    Probability observation -2.0 is from right distribution : 0.0


## Dimensionality Reduction##

Fitting and overfitting get worse with ''curse of dimensionality'' Bellman 1961

Think about a hypersphere. Its volume is  given by

\begin{equation}
  V_D(r) = \frac{2r^D\pi^{D/2}}{D\  \Gamma(D/2)},
\end{equation}

where $\Gamma(z)$ is the complete gamma function, $D$ is the dimension, and $r$ the radius of the sphere.

<!-- TEASER_END -->

If you populated a hypercube of size $2r$ how much data would be enclosed by the hypersphere
- as $D$ increases the fractional volume enclosed by the hypersphere goes to 0!

For example: the SDSS comprises a sample of 357 million sources.
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size..) and normalizing the data range $-1$ to $1$

probability of having one of the 357 million sources reside within a unit hypersphere 1 in 1.4$\times 10^5$.

```python
%matplotlib inline
import numpy as np
import scipy.special as sp
from matplotlib import pyplot as plt

def unitVolume(dimension, radius=1.):
    return 2*(radius**dimension *np.pi**(dimension/2.))/(dimension*sp.gamma(dimension/2.))

dim = np.linspace(1,100)

#------------------------------------------------------------
# Plot the results
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(dim,unitVolume(dim)/2.**dim)

ax.set_yscale('log')
ax.set_xlabel('$Dimension$')
ax.set_ylabel('$Volume$')

plt.show()

```

```python
D=30;unitVolume(D)/(2.**D)
```

    2.0410263396641429e-14

## Increasing interest in dimensionality reduction

## What can we do to reduce the dimensions##

**Principal component analysis or the Karhunen-Loeve transform**

The first refuge of a scoundrel...

Points are  correlated along a particular direction which doesn't align with the initial choice of axes.
- we should rotate our axes to align with this correlation.
- rotation preserves the relative ordering of data

Choose  rotation to maximize the ability to discriminate between the data points
*   first axis, or <u>principal component</u>, is direction of maximal variance
*   second principal component is orthogonal to the first component and maximizes the residual variance
*   ...

## Derivation of principal component analyses##

Set of data $X$: $N$ observations by $K$ measurements

Center data by subtracting the mean **Why?**

The covariance is

>$
C_X=\frac{1}{N-1}X^TX,
$

$N-1$ as the sample covariance matrix

We want a projection, $R$,  aligned with the directions of maximal variance ($Y= X R$) with covariance

>$
C_{Y} = R^T X^T X R = R^T C_X R
$

Derive  principal component by maximizing its variance (using Lagrange multipliers and constraint)

> $
\phi(r_1,\lambda_1) = r_1^TC_X r_1 - \lambda_1(r_1^Tr_1-1).
$

derivative of $\phi(r_1,\lambda)$ with respect to $r_1$ set to 0

> $
C_Xr_1 - \lambda_1 r_1 = 0.
$

$\lambda_1$ is the root of the equation $\det(C_X -
\lambda_1 {\bf I})=0$ and the largest eigenvalue

>$
\lambda_1 =  r_1^T C_X r_1
$

Other  principal components  derived by
applying additional constraint that components are uncorrelated (e.g., $r^T_2 C_X r_1 = 0$).

## Lagrangian mulitpliers

## Computation of principal components##

Common approach is eigenvalue decomposition of the covariance or correlation matrix,
or singular value decomposition (SVD) of the data matrix

** SVD given by**

>$
U \Sigma V^T = \frac{1}{\sqrt{N - 1}} X,
$

columns of $U$ are  _left-singular vectors_

columns of $V$ are the _right-singular vectors_

The columns of $U$ and $V$ form orthonormal bases ($U^TU = V^TV = I$)

Covariance matrix is

> $
\begin{eqnarray}
  C_X &=& \left[\frac{1}{\sqrt{N - 1}}X\right]^T \left[\frac{1}{\sqrt{N - 1}}X\right]\nonumber\\
      &=& V \Sigma U^T U \Sigma V^T\nonumber\\
      &=& V \Sigma^2 V^T.
\end{eqnarray}
$

right singular vectors $V$ are the principal components. We can calculate principal components from the SVD of $X$ - we dont need $C_X$.

Singular value decomposition (SVD) can factorize an N x K matrix into $U \Sigma V^T$. There are different conventions for computing the SVD in the literature, and this figure illustrates the convention used in this text. The matrix of singular values $\Sigma$ is always a square matrix of size [R x R] where R = min(N, K). The shape of the resulting U and V matrices depends on whether N or K is larger. The columns of the matrix U are called the left-singular vectors, and the columns of the matrix V are called the right-singular vectors. The columns are orthonormal bases, and satisfy $U^T U = V^T V = I$.

## Preparing data for PCA##

- Center data by subtracting the mean of each dimension
- For heterogeneous data (e.g., galaxy shape and flux) divide by  variance (whitening). **why?**
- For spectra or images normalize each row so integrated flux of each object is one.

```python
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA
from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA
def compute_PCA(n_components=5):
#    np.random.seed(500)
    nrows = 500
    ind = np.random.randint(spectra.shape[0], size=nrows)

    spec_mean = spectra[ind].mean(0)
#    spec_mean = spectra[:50].mean(0)

    # PCA: use randomized PCA for speed
    pca = PCA(n_components - 1)
    pca.fit(spectra[ind])
    pca_comp = np.vstack([spec_mean,
                          pca.components_])
    evals = pca.explained_variance_ratio_

    return pca_comp, evals

n_components = 5
decompositions, evals = compute_PCA(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(n_components):
    ax = fig.add_subplot(n_components, 2, 2*j+2)

    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, decompositions[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

## Interpreting the PCA##

Reconstruction of spectrum, ${x}(k)$, from the
eigenvectors, ${e}_i(k)$

>$ \begin{equation}
  {x}_i(k) = {\mu}(k) + \sum_j^R \theta_{ij} {e}_j(k),
\end{equation}
$

Truncating this expansion (i.e., $r<R$)

>$\begin{equation}
{x}_i(k) = {\mu}(k) + \sum_i^{r<R} \theta_i {e}_i(k),
\end{equation}
$

- eigenvectors ordered by their associated eigenvalues
- eigenvalues reflect variance  within each eigenvector (sum of the eigenvalues is total variance of the system).
- project a each spectrum onto these first few eigenspectra is a compression of the data

This is the sense in which PCA gives for dimensionality reduction.

```python
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA, KernelPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

## Comparing dimensionality reduction (PCA vs LLE)

*  Comparisons between the eigenvectors derived from PCA and known spectral types of galaxies
have shown that these statistically orthogonal components correlate
strongly with specific physical properties
*  given the cumulative nature of the sum of
variances used in PCA, astrophysically interesting components within
the spectra (e.g., sharp spectral lines or transient features for
certain galaxy populations) may not be reflected in the largest PCA
components.
- sum of
linear components can efficiently reconstruct the features within the
data does not always hold.  An example of this is the variation in
broad emission lines (such as those from quasars). The
variation in line width is an inherently nonlinear process and can
require a large number of components to fully characterize:
for broad line quasars over 30
components are required to reproduce the underlying spectra compared
to the 10 required for quiescent and star-forming galaxies.

## PCA with missing data##

Observed spectrum, $x^o$ is the true spectrum, ${x}$ plus a wavelength-dependent weight, ${w}$.

Weight is zero where data are missing and $1/{\sigma}^2$ for rest

Minimizing the quadratic deviation between  ${x}^o$ truncated reconstruction, $\sum_i \theta_i {e}_i$
and solving for $\theta_i$ gives

>$\begin{equation}
	\sum_k \theta_i {w}(k) {e}_i(k) {e}_j(k) =
	\sum_k {w}(k) {x}^o(k) {e}_j(k),
\end{equation}
$

If $M_{ij} = \sum_k {w}(k) {e}_i(k) {e}_j(k)$ and $F_i = \sum_k {w}(k) {x}^o(k) {e}_i(k)$ then

>$\begin{equation}
	\theta_i = \sum_j M_{ij}^{-1} F_{j},
\end{equation}
$

- $F_j$ are  coefficients derived from  gappy data
- $M_{ij}^{-1}$ shows how correlated  eigenvectors are over the missing regions.

An estimate of the uncertainty on the
reconstruction coefficients is given by

>$\begin{equation}
%Cov(\theta_i,\theta_j) = \frac{1}{N}M_{ij}^{-1}
{\rm Cov}(\theta_i,\theta_j) = M_{ij}^{-1}.
\end{equation}
$

Accuracy of this reconstruction will depend on the distribution of
the gaps within the data vector.

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import ticker

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(8, 10))
fig.subplots_adjust(hspace=0)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k', lw=2,
            label='True spectrum')

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'nev=%i' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1, prop=dict(size=14))

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()
```


<h1 style='font-size:4rem;color:orange;'>Math 267 Project #2

---

Solution notebook
---

## Euler's method for system of odes.

We consider the first-order autonomous system

$$ x'(t) = f (x, y)$$
$$ y'(t) = g(x, y) $$
along with the initial conditions

$$x(t_0) = x_0 $$
$$y(t_0) = y_0 $$

Notice that for the autonomous system t is not present in the slope functions f and g. The vector valued function $ F = (f (x, y), g(x, y))$ is called a vector field. We calculate the Euler approximation for this system by iterating the formulas

$$ t_{n+1} =t_n + Δt $$
$$ x_{n+1} =x_n + f(x_n, y_n) \Delta t$$
$$ y_{n+1} = y_n + g(x_n, y_n) \Delta t$$

---

## Exercise 1.  We will demonstrate how Euler's method can be used to solve the predator prey model and graph your results. The model is below.

$$x' = -0.1  x + 0.02 x y $$
$$y' = 0.2 y - 0.025xy$$
$$ x(0) = y(0) =6 $$

Note that x(t) represents the fox population at time t and y(t) represents the rabbit population. The populations are measured in thousands. Please see problem 3.3.11 in your text.

### Execute the cell below to import the necessary libraries

```python
# import libraries

import numpy as np
import matplotlib.pyplot as plt
# uncomment the line below if you are running a macbook

%config InlineBackend.figure_format ='retina'
```

<h3> The python code to numerically solve the above system is in the cell below. Read the code and understand each instruction.
    Afterwards execute the cell.

```python
# define the slope functions

def f(x,y):
    return -0.1*x + 0.02*x*y

def g(x,y):
    return 0.2*y - 0.025*x*y

h = 0.1  # set delta t
t = np.arange(0,100,h)

# Initialeze arrays to store the results.

x = np.zeros_like(t)
y = np.zeros_like(t)

# Set initial conditions:

x[0] = y[0] = 6

# implement Euler's method

for i in range (len(t)-1):
    x[i+1] = x[i] + f(x[i],y[i]) * h
    y[i+1] = y[i] + g(x[i],y[i]) * h

```

### Execute the cell below to graph your results.
You should note that the populations are changing periodically

```python
plt.figure(figsize=(8,5))
plt.plot(t,x,label='foxes')
plt.plot(t,y,label='rabbits')
plt.legend()
plt.grid()
plt.xlabel('time')
plt.ylabel('population');
```

### Execute the cell below to see the trajectory in the phase plane for this problem.  See chapter 10 in the text for more details on phase potraits.

```python
plt.figure(figsize=(8,5))
plt.plot(x,y,linewidth=2)
plt.xlim(-2,12)
plt.ylim(-2,12)
plt.grid()
plt.xlabel('foxes')
plt.ylabel('rabbits');
```

---
## Exercise 2. Improved Euler’s Method for Systems

From the discussion above we can readily see how to apply the Improved Euler’s method to the first order autonomous system of equations. All that is required is that we iterate the following formulas.

$$t_{n+1} =t_n + Δt$$
$$F1 =f(x_n,y_n)$$
$$G1 = g(x_n, y_n)$$
$$F2 =f(x_n +F1*Δt, y_n +G1*Δt)$$
$$G2 =g(x_n +F1*Δt, y_n +G1*Δt)$$
$$x_{n+1} =x_n + (F1 +F2)/2*Δt$$
$$y_{n+1} =y_n +(G1 + G2)/2*Δt$$

Redo the exercise above now using the improved Euler's method.  With the improved method h = 1 will work and you only need to compute 100 values.  So the following command will create your t array. Cut and past the cells above to recreate the plots.  You will know your solution is correct if your phase potrait shows a closed loop instead of a spiral.

```python
t = np.arange(0,100,1.0)

```

## Improved Euler Method

```python
# define the slope functions

def f(x,y):
    return -0.1*x + 0.02*x*y

def g(x,y):
    return 0.2*y - 0.025*x*y

h = 1.0  # set delta t
t = np.arange(0,100,h)

# Initialeze arrays to store the results.

x = np.zeros_like(t)
y = np.zeros_like(t)

# Set initial conditions:

x[0] = y[0] = 6

# implement Euler's method

for i in range (len(t)-1):

    F1 = f(x[i],y[i])
    G1 = g(x[i],y[i])

    F2 = f(x[i]+F1*h,y[i]+G1*h)
    G2 = g(x[i]+F1*h,y[i]+G1*h)

    slope1 = (F1+F2)/2
    slope2 = (G1+G2)/2

    x[i+1] = x[i] + slope1 * h
    y[i+1] = y[i] + slope2 * h

```

```python
plt.figure(figsize=(8,5))
plt.plot(t,x,label='foxes')
plt.plot(t,y,label='rabbits')
plt.legend()
plt.grid()
plt.xlabel('time')
plt.ylabel('population');
```

```python
plt.figure(figsize=(8,5))
plt.plot(x,y,linewidth=2)
plt.xlim(-2,12)
plt.ylim(-2,12)
plt.grid()
plt.xlabel('foxes')
plt.ylabel('rabbits');
plt.plot(6,6,'o',label="initial condition")
plt.legend()
```

---
### Answer the questions below.  Edit the cell and enter your answers.
---

1. The popuation for the foxes is periodic. Determine the period.

2. Find the value of t for t>0 where the populations are first equal.

3. Find all equilibrium solutions for the predator prey model.  Hint: set the vector field equation to zero and solve for x and y.  There are two equilibrium solutions ( points in the plane where the "cork" will not move.)

### 1. Find the period.

```python
vals=np.sort(x)[-2:] # find the two peak values
a=np.where(x==vals[0]) # find location of these peaks
b=np.where(x==vals[1])
period = int(abs(a[0]-b[0]))

print(f"The period for the population is {period}.")
```

    The period for the population is 45.

### 2. Find when the populations are equal.

```python
val=np.argmin(abs((x-y))[1:20])
print(f" The populations are first equal for t>0 at t={val+1}.")

```

     The populations are first equal for t>0 at t=6.

### 3. Find the equilibrium points.

We use sympy to solve the nonlinear system. For documentation on sympy go to https://docs.sympy.org/latest/index.html

Below we show the two equilbrium points are $$(0,0)\; \text {and}\; (8,5).$$

```python
import sympy as sym
sym.init_printing()

x,y = sym.symbols('x,y')
eq1 = sym.Eq(-0.1*x + 0.02*x*y,0)
eq2 = sym.Eq(0.2*y - 0.025*x*y,0)
result = sym.solve([eq1,eq2],(x,y))
print(result)
```

    [(0.0, 0.0), (8.00000000000000, 5.00000000000000)]


# Python Lecture 1: On Numbers, Variables, and Functions

### Lecture Notes by Jakov Ivan S. Dumbrique (jdumbrique@ateneo.edu)

MATH 100.2: Topics in Financial Mathematics II \
First Semester, S.Y. 2021-2022 \
Ateneo de Manila University

Today, we will learn four topics in Python:

0. How to put comments in your code
1. Numbers
2. Variables
3. Functions

## Section 0: Comments in Python

While your machine's Python compiler will skip comments when it encounters comments, writing comments in your code is still important in order to make your code readable by humans (and your future self).

```python
# Single line comments start with a number symbol.
"""
    Multiline strings can be written using three "s, and are often used as documentation.
"""

```

## Section 1: Numbers

Numbers are a primitive datatype in Python. Numbers can either be integers (int) or float.

```python
# You have numbers
3  # => 3

# Math operations are what you would expect
1 + 1   # => 2
8 - 1   # => 7
10 * 2  # => 20
35 / 5  # => 7.0

# Integer division rounds down for both positive and negative numbers.
5 // 3       # => 1
-5 // 3      # => -2
5.0 // 3.0   # => 1.0 # works on floats too
-5.0 // 3.0  # => -2.0

# The result of division is always a float
10.0 / 3  # => 3.3333333333333335

# Modulo operation
7 % 3   # => 1

# Exponentiation (x**y, x to the yth power)
2**3  # => 8

# Enforce precedence with parentheses
1 + 3 * 2    # => 7
(1 + 3) * 2  # => 8
```

    8

```python
# Comparison operators

# to check equality
1 == 1 # => True
1 == 1.0 # => True

# to check inequality
(3 - 2) != 0 # => True

# other comparison operators
(3 - 2) > 1 # => False
(3 - 2) >= 1 # => True
(3 ** 2) + (4 ** 2) < 5 ** 2 # => False
(3 ** 2) + (4 ** 2) <= 5 ** 2 # => True
```

    True

Definition: Simple Interest

Suppose that an amount $P$ is invested for $T$ years at an interest rate of $r$ per annum (p.a.). If the interest rate is a **simple rate**, then the terminal value of the investment is
\begin{align}
    A &= P + \underbrace{Pr + Pr + \cdots + Pr}_{T \text{ years}} \nonumber \\
    &= P(1 + \underbrace{r + \cdots + r}_{T \text{ years}}) \nonumber \\
    &= P(1+rT). \label{Simple Interest}
\end{align}

Example 1

Suppose that $\$408$ is deposited in a newly created savings account that pays a simple interest of $3.435\%$ per annum. If no deposits or withdrawals are made on the account since the aforementioned transaction, how much is withdrawn if the account is closed and emptied in $6$ months?

$A = \$408\left[1+ 3.435\% \left(\dfrac{6}{12}\right)\right] \approx \boxed{\$415.01}.$

```python
408*(1+0.03435*(6/12))
```

    415.00739999999996

## Section 2: Variables

Variables provide a way to associate names with Python objects. We use variables in order to reuse names instead of writing explicitly the values repeatedly.

We use an equal sign (=) to assign a value to a variable.

```python
pi = 3.14159
```

```python
pi
```

    3.14159

Variable names
1. Can contain uppercase and lowercase letters, digits (but they cannot start with a digit), and the special character _
2. Are case-sensitive! (pi is different from Pi)
3. Cannot be [Python reserved words (or keywords)](https://www.w3schools.com/python/python_ref_keywords.asp)

Example 2: Assign variables to Example 1

Suppose that $\$408$ is deposited in a newly created savings account that pays a simple interest of $3.435\%$ per annum. If no deposits or withdrawals are made on the account since the aforementioned transaction, how much is withdrawn if the account is closed and emptied in $6$ months?

```python
P = 408
r = 0.03435
T = 6/12

P * (1 + r*T)
```

    415.00739999999996

## Section 3: Functions

1. if you will reuse a piece of code, you should write a function for it
2. functions are not run in a program until they are “called” or “invoked” in a program
3. characteristics of a function:
    - has a name
    - has parameters (0 or more)
    - has a docstring (optional but recommended)
    - has a body
    - returns something

Example 3: Write a function for Example 2

Suppose that $\$408$ is deposited in a newly created savings account that pays a simple interest of $3.435\%$ per annum. If no deposits or withdrawals are made on the account since the aforementioned transaction, how much is withdrawn if the account is closed and emptied in $6$ months?

```python
def get_future_value_simple(P, r, T):
    """
    Input: principal P, simple interest rate r p.a., investment time period T in years
    Returns the future value of a principal P invested for T years at a simple rate r
    """
    return P * (1 + r*T)
```

```python
get_future_value_simple(408, 0.03435, 0.5)
```

    415.00739999999996

```python
get_future_value_simple(P=408, r=0.03435, T=0.5)
```

    415.00739999999996

```python
P = 408
r = 0.03435
T = 6/12
get_future_value_simple(P, r, T)
```

    415.00739999999996

Tip:  use functions if you will reuse a chunck of code. Think of a function as a black box that hides tedious coding details.

Example 4: Reusing Functions

Suppose you invest $\$500$ in an account that pays a simple interest of $5.75\%$ p.a. How much will it be worth in $1.5$ years? $20$ years?

```python
get_future_value_simple(P=500, r=0.0575, T=1.5)
```

    543.125

```python
get_future_value_simple(P=500, r=0.0575, T=20)
```

    1075.0000000000002

Definition: Compound Interest

Suppose that an amount $P$ is invested for $T$ years at an interest rate of $r$ per annum (p.a.). If the interest rate is **compounded** $\boldsymbol{m}$ **times a year**, then the terminal value of the investment is
\begin{align}
    A = P\left(1+\dfrac{r}{m}\right)^{mT}. \label{Compounding Interest}
\end{align}

Exercise

Suppose you invest $\$500$ in an account that pays a compound interest of $5.75\%$ p.a. How much will it be worth in $1.5$ years if the interest rate is
1. compounded annually?
2. compounded semiannually?
3. compounded quarterly?
4. compounded monthly?

Write a function and reuse that function to answer the above question.

```python
def get_future_value_comp(P, r, m, T):
    """
    Input: principal P, interest rate r p.a. compounded m times a year, investment time period T in years
    Returns the future value of a principal P invested for T years at an
    interest rate r p.a. compounded m times a year
    """
    return P * (1 + r/m)**(m*T)
```

```python
P = 500
r = 0.0575
T = 1.5
get_future_value_comp(P, r, 1, T)
```

    543.739105494308

```python
get_future_value_comp(P, r, 2, T)
```

    544.3767255859376

```python
get_future_value_comp(P, r, 4, T)
```

    544.7048313758173

```python
get_future_value_comp(P, r, 12, T)
```

    544.9271497450427

```python

```


# Lab 3
## Introduction
In this lab we will analyse population dynamics under the logisitic model with managed harvesting.

First import the modules we need.

```python
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
from numpy import meshgrid, linspace, sqrt, arange
from scipy.integrate import odeint
```

## Harvesting of fish
A population of fish in a lake, left to its own devices, is modelled by the logistic differential equation
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y),
\end{align}
where the population $y$ is in units of thousands of fish and time $t$ is measured in years.

First define a function for $\mathrm{d}y/\mathrm{d}x$ in terms of $y$ and $x$.

```python
def diff_eq(y, x):
    return 4 * y * (1 - y)
```

Next define a function that creates a Plotly Figure object that contains a slope field and, optionally, a few solutions to initial value problems.

It automates a few things we did in the last lab.

- `diff_eq` is the differential equation to be plotted
- `x` and `y` should be outputs from `meshgrid`.
- `args` is any additional arguments to `diff_eq` (we will use that below).
- `initial_values` is a list (or array) of starting $y$ values from which approximate solutions will start. The corresponding $x$ value is the minimum element of `x`.

Note that the numerical solutions will plotted for the whole range of $x$ values in `x`, so if they blow up you will probably get a warning and less-than-useful plot.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    fig, ax = plt.subplots(figsize=(5, 5))
    q = ax.quiver(x, y, 1/L, S/L, scale=25, headwidth=0, headlength=0, color='grey')
    x = linspace(x.min(), x.max())
    data = {'x': x}
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args)[:,0]
        data[f'y({x[0]}) = {y0}'] = y
    data = pd.DataFrame(data)
    data = data.melt(id_vars=['x'], value_name='y', var_name='initial value')
    sns.lineplot(data=data, x='x', y='y', hue='initial value')
    return fig, ax
```

The slope field below should hopefully give you some idea for the fish population dynamics.

Note that we use `arange` rather than `linspace` this week so that we can carefully control the increments between our grid points. `arange(0, 1.1, 0.25)` returns an array that starts with 0 and increments by 0.25 until it exceeds 1.1.

The plot also contains the solution curves for
$y(0) = 1$ and $y(0) = 0.4$. Edit the cell to also include the solution curve for $y(0)=1.4$.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig, ax = create_slope_field(diff_eq, x, y, initial_values=(0.4, 1))
```

### Equilibrium solutions
Looking back to our differential equation, $\mathrm{d}y/\mathrm{d}t = 0$ when $y(t) = 0$ or $y(t) = 1$. Looking at the slope field, we see that the equilibrium solution $y(t) = 1$ is stable (this is the carrying capacity here, corresponding to 1000 fish), whereas the equilibrium solution $y(t) = 0$ is unstable. Any non-zero initial population will eventually stabilise at 1000 fish.

### What will happen if harvesting is now commenced at a steady rate?
For the simplest harvesting model, assume that $H$ units (thousands) of fish are taken
continuously (smoothly) over the year, rather than at one instant each year.
Note that the units of $H$ are the same as those of $\mathrm{d}y/\mathrm{d}t$, thousands of fish per year, so we simply subtract $H$ from the RHS of our existing equation to give the DE with harvesting as
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y) - H.
\end{align}
Again, the (constant) equilibrium solutions are found by setting $\mathrm{d}y/\mathrm{d}t = 0$, giving from the quadratic formula (check this),
\begin{align}
y(t) = \frac{4\pm\sqrt{16-16H}}{8} = \frac{1\pm\sqrt{1-H}}{2}.
\end{align}
What happens after harvesting starts will depend on the equilibrium solutions, their
stability and the initial number of fish $y(0)$.

Start by redefining `diff_eq` to include the `H` parameter. Note that defining `diff_eq` again overides our original definition.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
```

Now set $H = 0.6$ and plot the slope field. This is done by setting `args=(0.6,)` when we call `create_slope_field`. This is exactly how you would pass additional arguments like this one to `odeint` if you were calling it directly.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig, ax = create_slope_field(diff_eq, x, y, args=(0.6,))
```

From the solutions to the quadratic equation above, the equilibrium solutions of the DE are found to be $y(t) \approx 0.184$ and $y(t) \approx 0.816$. The previous equilibrium solution with no harvesting at $y(t) = 0$ has moved up to $y(t) \approx 0.184$, while the previous equilibrium solution with no harvesting at $y(t) = 1$ has moved down to $y(t) \approx 0.816$.

From the slope field, we see that the equilibrium solution $y(t) \approx 0.184$ is unstable, whereas the equilibrium solution $y(t) \approx 0.816$ is stable. If the population ever falls below about 0.184, or 184 fish, it will then drop to 0. This is a new feature, introduced by harvesting.

In the cell below, use `create_slope_field` to experiment by plotting the solutions to the initial value problems $y(0)=0.183$ and $y(0)=0.25$. Extend the $x$ range of your slope field until the top line is close to equlibrium. Note that if you extend it too far you will break `odeint` (why?). You may also like to increase the increments in `arange` to make the plot clearer.

```python

```

## Exercises

In the following questions we will experiment with how different levels of harvesting affect outcomes for the fish population.

This week the questions will be a combination of plots and written answers. Written answers go in blank cells, but if a cell says "Type _Markdown_ and LaTeX: $\alpha^2$", you should double-click on it to edit it.

1. Assume that the harvest is now 600 fish per year. **On the same figure,**
  a. plot the slope field,
  b. plot the equilibrium solutions that we found in above, and
  c. plot the solution curves for $y(0)=1$, $y(0)=0.3$, and $y(0)=0.15$.

```python

```

1.  d. In the cell below, describe the behaviour of the fish population for each of these five initial numbers of fish.

2. a. i. Assume that $H=0.8$. Plot the slope field and five solutions, one for each equilibrium solution and one for each region between, above, or below them. You can use the equation from the lab to calculate the equilibrium solutions.

```python

```

2. a. ii. In the cell below, describe the limiting behaviour of each line.

2. b. i. Assume that $H=1$. Plot the slope field and three solutions for the equilibrium solution and the regions above and below it.

```python

```

2. b. ii. Describe the limiting behaviour of each line.

2. c. i. Assume that  𝐻=1.2. Plot the slope field and two or three solutions.

```python

```

2. c. iii. Describe the limiting behaviour of the lines.

3. Use `sns.lineplot` to plot a bifurcation graph showing the reulationship between $H$ and the locations of the critical points $c$. $H$ should go on the horizontal axis but you may find it easier to calculate $H$ in terms of $c$ by rewriting the above equations giving the equilibrium solutions.

  Hint: your plots won't work unless you add `sort=False` as a keyword argument to `sns.lineplot`, and you may observe some artifacts in your plot unless you also set `estimator=None`.

```python

```

4. Summarize what happens to the equilibrium solutions and their stability as $H$
is increased from 0 to beyond 1. Refer to your plots to support your answers.

5. What is a reasonable strategy for sustainable fishing in this case?
Don’t forget to allow qualitatively for minor catastrophes, such as disease or temporary overfishing.


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

## Curve fitting

\begin{equation}
RMSE=\sqrt{\frac{1}{n}\sum{(y-\hat{y})^2}}\\
R^2=1-\frac{\sum{(y-\hat{y})^2}}{\sum{(y-\bar{y})^2}}
\end{equation}

## Example:

### 1.Fit gas equation
\begin{equation}
PV^\gamma=constant(K)\\
logP=logK-\gamma logV\\
y=a+bx
\end{equation}

```python
df = pd.read_excel('data/gas.xlsx')
df
```

      volume(cm3)
      pressure(gm/cm2)

      0
      54.3
      61.2

      1
      61.8
      49.2

      2
      72.4
      37.6

      3
      88.7
      28.4

      4
      118.6
      19.2

      5
      194.0
      10.1

```python
plt.figure(figsize = [6,6])
plt.scatter(df['volume(cm3)'],df['pressure(gm/cm2)'])
plt.xlabel('volume(cm3)')
plt.ylabel('pressure(gm/cm2)')
plt.title('Gas')
plt.show()
```

```python
n=6
x=np.log(df['volume(cm3)'])
y=np.log(df['pressure(gm/cm2)'])
z=np.polyfit(x, y, 1) # linear  fit =y=a+bx
p=np.poly1d(z)  #  [a,b]
ey=p(x) # expected value
a="{:.3}".format(p[0])
b="{:.3}".format(p[1])
K=np.exp(p[0])
gamma=p[1]

RMSE=np.sqrt(np.sum((y-ey)**2)/n)
R_square=1-(np.sum(np.sum(np.sum((y-ey)**2))/np.sum((y-np.mean(y))**2)))
rmse="{:.2}".format(RMSE)
r2="{:.2}".format(R_square)

plt.figure(figsize = [6,6])
plt.plot(x, y, 'ro',label='observed')
plt.plot(x, ey, 'b-',label='fitted, a= '+str(a)+', b='+str(b))
plt.xlabel('log(V)')
plt.ylabel('log(P)')
plt.figtext(0.2, 0.3, 'RMSE = '+str(rmse)+',r2='+str(r2))
plt.legend()
plt.title('Fitted curve')
plt.show()
print('gamma =',"{:.2}".format(-gamma))
print('K=',"{:.2}".format(K))

```

### 2.Fit
\begin{equation}
f(\theta)=A+B\cos\theta+C\cos^2\theta\\
y=A+Bx+Cx^2
\end{equation}

```python
df1 = pd.read_excel('data/scattering.xlsx')
df1
```

      theta(degree)
      f(theta)

      0
      30
      11

      1
      45
      13

      2
      90
      16

      3
      120
      17

      4
      150
      14

```python
plt.figure(figsize = [6,6])
plt.scatter(df1['theta(degree)'],df1['f(theta)'])
plt.xlabel(r'$\theta$(degree)')
plt.ylabel(r'f($\theta$)')
plt.title('Scattering')
plt.show()
```

```python
n1=5
x1=np.cos(df1['theta(degree)']*np.pi/180)
y1=df1['f(theta)']
z1=np.polyfit(x1, y1, 2) # second order  fit y=a+bx+cx2
p1=np.poly1d(z1)  #  [c,b,a]
ey1=p1(x1) #expected value
A="{:.3}".format(p1[0])
B="{:.3}".format(p1[1])
C="{:.3}".format(p1[2])

RMSE1=np.sqrt(np.sum((y1-ey1)**2)/n1)
R_square1=1-(np.sum(np.sum(np.sum((y1-ey1)**2))/np.sum((y1-np.mean(y1))**2)))
rmse1="{:.2}".format(RMSE1)
r21="{:.2}".format(R_square1)

plt.figure(figsize = [6,6])
plt.plot(x1, y1, 'ro',label='observed')
plt.plot(x1, ey1, 'b-',label='fitted, A= '+str(A)+', B='+str(B)+',C='+str(C))
plt.xlabel(r'cos($\theta$)')
plt.ylabel(r'f($\theta$)')
plt.figtext(0.2, 0.3, 'RMSE = '+str(rmse1)+',r2='+str(r21))
plt.legend()
plt.title('Fitted curve')
plt.show()

```

### 3.Determination of decay conctant and half life
\begin{equation}
N=N_0~e^{-\lambda~t}\\
logN=logN_0-\lambda~t\\
y=A+BX
\end{equation}

```python
df2 = pd.read_excel('data/env-dec.xlsx')
df2
```

      t [s]
      N [s-1]

      0
      0
      32

      1
      5
      28

      2
      10
      29

      3
      15
      28

      4
      20
      25

      ...
      ...
      ...

      78
      390
      1

      79
      395
      1

      80
      400
      2

      81
      405
      2

      82
      410
      1

83 rows × 2 columns

```python
plt.figure(figsize = [6,6])
plt.scatter(df2['t [s]'],df2['N [s-1]'])
plt.xlabel('time(s)')
plt.ylabel('Count($s^{-1}$)')
plt.title('Decay curve')
plt.show()
```

```python
n2=83
x2=df2['t [s]']
y2=np.log(df2['N [s-1]'])
z2=np.polyfit(x2, y2, 1) # second order  fit y=a+bx
p2=np.poly1d(z2)  #  [c,b,a]
ey2=p2(x2) #expected value
A1="{:.3}".format(p2[0])
B1="{:.3}".format(-p2[1])

RMSE2=np.sqrt(np.sum((y2-ey2)**2)/n2)
R_square2=1-(np.sum(np.sum(np.sum((y2-ey2)**2))/np.sum((y2-np.mean(y2))**2)))
rmse2="{:.2}".format(RMSE2)
r22="{:.2}".format(R_square2)

plt.figure(figsize = [6,6])
plt.plot(x2, y2, 'ro',label='observed')
plt.plot(x2, ey2, 'b-',label='fitted, A= '+str(A1)+', B='+str(B1))
plt.xlabel('time(sec)')
plt.ylabel('ln(A)')
plt.figtext(0.2, 0.3, 'RMSE = '+str(rmse2)+',r2='+str(r22))
plt.legend()
plt.title('Fitted curve')
plt.show()
```

```python
print('decay constant =%0.6f /sec'%-p2[1])
```

    decay constant =0.008246 /sec

```python
T=-0.693/p2[1]
print('half life = %0.2f sec'%T)
```

    half life = 84.04 sec

### 4.Absorption of coefficient
\begin{equation}
N=N_0~e^{-\mu~x}\\
logN=logN_0-\mu~x\\
y=A+BX
\end{equation}

```python
df3 = pd.read_excel('data/env-beta.xlsx')
df3
```

      n
      N [s-1]

      0
      0
      8.0

      1
      1
      6.0

      2
      2
      5.0

      3
      3
      4.0

      4
      4
      3.0

      5
      5
      2.0

      6
      6
      2.0

      7
      7
      2.0

      8
      8
      2.0

      9
      9
      1.0

      10
      10
      1.0

      11
      11
      1.0

      12
      12
      0.5

      13
      13
      0.5

      14
      14
      0.5

      15
      15
      0.5

      16
      16
      1.0

      17
      17
      0.5

      18
      18
      0.4

      19
      19
      0.3

      20
      20
      0.1

```python
plt.figure(figsize = [6,6])
plt.scatter(0.03*df3['n'],df3['N [s-1]'])# one sheet of Al is 0.03 mm thick
plt.xlabel('Thickness(mm)')
plt.ylabel('Count($s^{-1}$)')
plt.title('Absorption curve')
plt.show()
```

```python
n3=20
x3=0.03*df3['n']
y3=np.log(df3['N [s-1]'])
z3=np.polyfit(x3, y3, 1) # second order  fit y=a+bx
p3=np.poly1d(z3)  #  [c,b,a]
ey3=p3(x3) #expected value
A2="{:.3}".format(p3[0])
B2="{:.3}".format(-p3[1])

RMSE3=np.sqrt(np.sum((y3-ey3)**2)/n3)
R_square3=1-(np.sum(np.sum(np.sum((y3-ey3)**2))/np.sum((y3-np.mean(y2))**2)))
rmse3="{:.2}".format(RMSE3)
r23="{:.2}".format(R_square3)

plt.figure(figsize = [6,6])
plt.plot(x3, y3, 'ro',label='observed')
plt.plot(x3, ey3, 'b-',label='fitted, A= '+str(A2)+', B='+str(B2))
plt.xlabel('Thickness(mm)')
plt.ylabel('ln(N)')
plt.figtext(0.2, 0.3, 'RMSE = '+str(rmse3)+',r2='+str(r23))
plt.legend()
plt.title('Fitted curve')
plt.show()
```

```python
print('absorption coefficient =%0.3f /mm'%-p3[1])
```

    absorption coefficient =5.695 /mm

```python
density=2.7 # in cgs
mu_m=-10*p3[1]/density # in cgs
print('mass absorption coefficient =%0.3f sq.cm/gm'%mu_m)
```

    mass absorption coefficient =21.091 sq.cm/gm

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import plotutils as pu
from sympy import *
%matplotlib inline
```

We add vectors to get $v + w$. We multiply them by numbers $c$ and $d$ to get $cv$ and $dw$. Combining those two operations (adding $cw$ to $dw$) gives the **linear combination** $cv + dw$.

$$cv + dw = c\begin{bmatrix}1\\1\end{bmatrix} + d\begin{bmatrix}2\\3\end{bmatrix} = \begin{bmatrix}c + 2d\\c + 3d\end{bmatrix}$$

We can also write a column vector *inline* as $w = (2, 3)$ but that is not the same as the row vector $\begin{bmatrix}2 & 3\end{bmatrix}$.

```python
fix, axes = plt.subplots(1, figsize=(3, 3))
pu.setup_axes(axes, xlim=(-1, 4), ylim=(-1, 4))
arrowprops = {'head_length': 0.25, 'head_width': 0.18}
plt.arrow(0, 0, 1, 1, **arrowprops, ec='r', fc='r')
plt.arrow(0, 0, 2, 3, **arrowprops, ec='b', fc='b')
```

## using sympy
Sympy is a wonderful toolkit for doing symbolic programming. We can also use it to *cheat* and help it solve systems of equations for us. Let's put it to work on one of the problems of the linear algebra book.

Write down the three equations for $c$, $d$ and $e$ so that $cu + dv + ew = b$ and try to find $c$, $d$ and $e$.

$$
u = \begin{bmatrix}2\\-1\\0\end{bmatrix}
v = \begin{bmatrix}-1\\2\\-1\end{bmatrix}
w = \begin{bmatrix}0\\-1\\2\end{bmatrix}
b = \begin{bmatrix}1\\0\\0\end{bmatrix}
$$

The equations are:

$$
\begin{cases}
2c - d &= 1\\
-c + 2d - e &= 0\\
-d + 2e &= 0\\
\end{cases}
$$

And we could try solve them ourselves which is not that hard to do via elimination but alternatively we can also use `sympy`.

```python
s = [S('2*c - d - 1'), S('-c + 2*d - e'), S('-d + 2*e')]
solve(s)
```

    {d: 1/2, c: 3/4, e: 1/4}

```python

```


# M2AA3 Chapter 2, Lesson 2 - Generating Orthogonal Polynomials

```python
#Setup for Sympy
import sympy as sp
```

We wish to construct a list of polynomial $\phi_n(x)$ which is orthogonal with respect to the inner product
$$\langle f(x),g(x) \rangle = \int_a^b w(x)f(x)g(x)dx$$
where $w(x) > 0$ is the weight function. Let $\|f(x)\|^2 = \langle f(x),f(x) \rangle$. We may use the following recurrence relation:
$$\phi_j(x) = \bigg(x - \frac{\langle x\phi_{j-1}(x),\phi_{j-1}(x) \rangle}{\|\phi_{j-1}(x)\|^2} \bigg)\phi_{j-1}(x) - \frac{\|\phi_{j-1}(x)\|^2}{\|\phi_{j-2}(x)\|^2}\phi_{j-2}(x), j \geq 1$$
where $\phi_{-1}(x)=0$ and $\phi_{0}(x)=1$. Therefore we have
$$\phi_{1}(x) = x - \frac{\langle x,1 \rangle}{\| 1 \|^2} = x - \frac{\int_a^b x w(x)dx}{\int_a^b w(x)dx}$$
Our aim is to define a function 'ortho', which inputs $w(x)$, $a$, $b$ and $r$ and output a list $\phi_0(x), ..., \phi_r(x)$.

Step 1 - Define inner product and norm

```python
x = sp.symbols('x')
w = sp.Function('w')
f = sp.Function('f')
g = sp.Function('g')
```

```python
def inner(w,a,b,f,g):
    output = sp.integrate(w*f*g, (x,a,b))
    return output

def norm(w,a,b,f):
    return sp.sqrt(inner(w,a,b,f,f))
```

Step 2 - Write the recursive function

```python
phi = sp.Function('phi')
phiminus1 = sp.Function('phiminus1')
phiminus2 = sp.Function('phiminus2')
```

```python
def ortho(w,a,b,r):
    phiminus2 = 1
    phiminus1 = x - inner(w,a,b,x,1)/(norm(w,a,b,1))**2
    if r == 0:
        return [phiminus2]
    elif r == 1:
        return [phiminus2,phiminus1]
    else:
        philist = [phiminus2,phiminus1]
        for i in range(r-1):
            phi = (x - inner(w,a,b,x*phiminus1,phiminus1)/(norm(w,a,b,phiminus1))**2)*phiminus1 - ((norm(w,a,b,phiminus1)/norm(w,a,b,phiminus2))**2)*phiminus2
            phi = sp.simplify(phi)
            philist.append(phi)
            phiminus2 = phiminus1
            phiminus1 = phi
    return philist
```

## Legendre Polynomials
Here is an example of list of polynomials which are orthogonal with $w(x) = 1$. These are called Legendre Polynomials.

```python
olist = ortho(1,-1,1,5)
print(olist)
```

    [1, x, x**2 - 1/3, x*(x**2 - 3/5), x**4 - 6*x**2/7 + 3/35, x*(63*x**4 - 70*x**2 + 15)/63]

Observe that they are very similar to the Chebyshev Polynomials. Recall that they are defined as followed:
$$L_j(x) = \frac{2j-1}{j} x L_{j-1}(x) - \frac{j-1}{j} L_{j-2}(x)$$
where $L_0(x) = 1, L_1(x) = x$. If $\phi_j(x)$ is the orthogonal polynomials obtained, then
$\phi_j(x) = \frac{L_j(x)}{a_j}, j \geq 1$, where $a_j$ is the leading coefficient of $L_j(x)$.

To verify this let's generate the first $r$ Legendre Polynomials.

```python
P = sp.Function('P')
Pminus1 = sp.Function('Pminus1')
Pminus2 = sp.Function('Pminus2')
```

```python
def Leb(r):
    Pminus2 = 1
    Pminus1 = x
    if r == 0:
        return [Pminus2]
    elif r == 1:
        return [Pminus2,Pminus1]
    else:
        Plist = [Pminus2,Pminus1]
        for i in range(2,r+1):
            P = (sp.Rational(2*i-1,i))*x*Pminus1 - sp.Rational(i-1,i)*Pminus2
            P = sp.simplify(P)
            Plist.append(P)
            Pminus2 = Pminus1
            Pminus1 = P
    return Plist
```

Then we verify that the polynomials are the same.

```python
Leblist = Leb(3)
print(Leblist)
Lebdivided = [1]+[sp.simplify(Leblist[i]/sp.polys.polytools.LC(Leblist[i])) for i in range(1,len(Leblist))]
print(Lebdivided)
plist = ortho(1,-1,1,3)
print(plist)
print(plist == Lebdivided)
```

    [1, x, 3*x**2/2 - 1/2, x*(5*x**2 - 3)/2]
    [1, x, x**2 - 1/3, x*(x**2 - 3/5)]
    [1, x, x**2 - 1/3, x*(x**2 - 3/5)]
    True

We may find the roots of polynomials.

```python
sp.solve(olist[3],x)
```

    [0, -sqrt(15)/5, sqrt(15)/5]

## Chebyshev Polynomials
Here is another example of list of polynomials which are orthogonal with $w(x) = \frac{1}{\sqrt{1 - x^2}}$.

```python
tlist = ortho(1/sp.sqrt(1-x**2),-1,1,3)
print(tlist)
```

    [1, x, x**2 - 1/2, x*(x**2 - 3/4)]

Observe that they are very similar to the Chebyshev Polynomials. Recall that they are defined as followed:
$$T_j(x) = 2x T_{j-1}(x) - T_{j-2}(x), j \geq 2$$
where $T_0(x) = 1, T_1(x) = x$. It could be proved that $T_j = \cos (j \cos^{-1} x)$.
If $\phi_j(x)$ is the orthogonal polynomials obtained, then $\phi_j(x) = \frac{T_j(x)}{2^{j-1}}, j \geq 1$. <br>

To verify this let's generate the first $r$ Chebyshev Polynomials.

```python
T = sp.Function('T')
Tminus1 = sp.Function('Tminus1')
Tminus2 = sp.Function('Tminus2')
```

```python
def Che(r):
    Tminus2 = 1
    Tminus1 = x
    if r == 0:
        return [Tminus2]
    elif r == 1:
        return [Tminus2,Tminus1]
    else:
        Tlist = [Tminus2,Tminus1]
        for i in range(r-1):
            T = 2*x*Tminus1 - Tminus2
            T = sp.simplify(T)
            Tlist.append(T)
            Tminus2 = Tminus1
            Tminus1 = T
    return Tlist
```

Then we verify that the polynomials are the same.

```python
Chelist = Che(3)
print(Chelist)
Chelistdivided = [1] + [sp.simplify(Chelist[i]/(2**(i - 1))) for i in range(1, len(Chelist))]
print(Chelistdivided)
tlist = ortho(1/sp.sqrt(1-x**2),-1,1,3)
print(tlist)
print(tlist == Chelistdivided)
```

    [1, x, 2*x**2 - 1, x*(4*x**2 - 3)]
    [1, x, x**2 - 1/2, x*(x**2 - 3/4)]
    [1, x, x**2 - 1/2, x*(x**2 - 3/4)]
    True

We may find the roots for the polynomials.

```python
sp.solve(tlist[2],x)
```

    [-sqrt(2)/2, sqrt(2)/2]

M2AA3/M2AA3-Least-Square/Lesson 02 Best Approximation Problem/Orthogonal Polynomial.ipynb