---
## 01. Data Analysis. Statistics Basics

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

### About this notebook

In this worksheet, we introduce some basic aspects and definitions of statistics for working with astrophysical data.

---

### Statistics with `numpy`

Statistics are designed to summarize, reduce or describe data. A statistic is a function of the data alone!

Consider a dataset $\{ x_1, x_2, x_3, ...\}$

Some important quantities defined to describe the dataset are **average** or **mean**, **median**, **maximum value**, **average of the squares**, etc.

Now, we will explore some of these concepts using the `numpy` package and a set of data taken from the book *Computational Physics* by Mark Newman that can be downloaded from

http://www-personal.umich.edu/~mejn/computational-physics/

The file `sunspots-since1749.txt` contains the observed number of sunspots on the Sun for each month since January 1749. In the file, the first column corresponds to the month and the second the sunspots number.

```python
import matplotlib.pyplot as plt
import numpy as np
```

```python
month, sunspots = np.loadtxt('sunspots-since1749.txt', unpack=True)
```

The number of samples in the dataset is

```python
print(month.size, ' months')
```

    3143  months

```python
years, months = divmod(month.size, 12)
print(years, ' years and ', months, ' months')
```

    261  years and  11  months

A scatter plot showing the behavior of the dataset,

```python
fig, ax = plt.subplots(figsize=(7,5))

ax.scatter(month, sunspots, marker='.')
ax.set_xlabel(r'months since january 1749')
ax.set_ylabel(r'number of sunspots')
plt.show()
```

and a plot showing the same dataset is

```python
fig, ax = plt.subplots(figsize=(7,5))

ax.plot(month, sunspots)
ax.set_xlabel(r'months since january 1749')
ax.set_ylabel(r'number of sunspots')
plt.show()
```

#### Maximum and Minimum

```python
np.ndarray.max(sunspots)
```

    253.8

```python
np.ndarray.min(sunspots)
```

    0.0

---
#### Mean and Weighted Average
The function `numpy.mean()` returns the arithmetic average of the array elements.

https://numpy.org/doc/stable/reference/generated/numpy.mean.html?highlight=mean#numpy.mean

\begin{equation}
 \text{mean} = \frac{\sum x_i}{N}
\end{equation}

The monthly mean number of spots in the time period of the data set is

```python
mean_sunspots = np.mean(sunspots)
mean_sunspots
```

    51.924498886414256

The function `numpy.average()` returns the weighted average of the array elements.

https://numpy.org/doc/stable/reference/generated/numpy.average.html?highlight=average#numpy.average

Including a weight function with values $w_i$, this function uses the formula

\begin{equation}
 \text{average} = \frac{\sum x_i w_i}{\sum w_i}
\end{equation}

```python
w = sunspots/np.ndarray.max(sunspots)
w
```

    array([0.2285264 , 0.24665091, 0.27580772, ..., 0.09929078, 0.09259259,
           0.08510638])

```python
np.average(sunspots, weights=w)
```

    89.74573872218345

If we do not include the weights or if all weights are equal, the average is equivalent to the mean

```python
np.average(sunspots)
```

    51.924498886414256

#### Median and Mode

The function `numpy.median()` returns the median of the array elements along a given axis.

Given a vector V of length N, the median of V is the middle value of a sorted copy of V, when N is odd, and the average of the two middle values of the sorted copy of V when N is even.

https://numpy.org/doc/stable/reference/generated/numpy.median.html

```python
np.median(sunspots)
```

    41.5

The mode corresponds to the value ocurring most frequently in the dataset and it can be seen as the location of the peak of the histogram of the data.

Although the `numpy` package does not have a mode function, we can use the function `scipy.mode()`to calculate the modal value.

```python
from scipy import stats

stats.mode(sunspots)
```

    ModeResult(mode=array([0.]), count=array([67]))

This result indicates that the mode is the number $0.$ and that it appears 67 times in the sunspots dataset.

A histogram may help to show the behavior of the data,

```python
fig, ax = plt.subplots(figsize=(7,5))

ax.hist(sunspots, bins=25)
ax.set_xlabel(r'number of sunspots')
ax.set_ylabel(r'')
plt.show()
```

```python
fig, ax = plt.subplots(figsize=(7,5))

ax.hist(sunspots, bins=400)
ax.set_xlabel(r'number of sunspots')
ax.set_ylabel(r'')
plt.show()
```

## Astrophysical Signals

In astrophysics, it is usual to detect signals immersed in noise.
For example, in radioastronomy, the emission of sources such as radio-galaxies, pulsars, supernova remnants, etc. are received by radio-telescopes in Earth. they detect emission at frequencies in the order of megahertz, which is similar to many radio stations and therefore, the signal is immersed in a lot of noise!

The flux density of these signals is measured in *Janskys* (Jy), which is equivalent to

\begin{equation}
1 \text{ Jansky} = 10^{-26} \frac{\text{Watts}}{\text{m}^2\text{ Hz}}
\end{equation}

Hence, flux is a measure of the spectral power received by a telescope detector of unit projected area.

Usually, astrophysical sources have flux densities much smaller than the noise around them.

| Source | Flux Density (Jy) |
|:--------|-------------------:|
|Crab Pulsar at 1.4GHz|$\sim 0.01$|
|Milky Way at 10 GHz|$\sim 2 \times 10^3$|
|Sun at 10 GHz|$\sim 4 \times 10^6$|
|Mobile Phone| $\sim 1.1 \times 10^8$|

Consider a synthetic signal with the Gaussian form,

```python
x = np.linspace(0, 100, 200)
y = 0.5*np.exp(-(x-40)**2./25.)

fig, ax = plt.subplots(figsize=(7,5))

ax.plot(x,y)
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'Flux Density (Jy)')
plt.show()

```

Now, we will add some random noise by defining two random arrays

```python
noise1 = np.random.rand(200)
noise2 = np.random.rand(200)

fig, ax = plt.subplots(figsize=(7,5))
ax.plot(x,noise1, color='darkblue')
ax.plot(x,noise2, color='cornflowerblue')

ax.set_xlabel(r'$x$')
ax.set_ylabel(r'noise')
plt.show()
```

We add these random noise arrays to the Gaussian profile to obtain

```python
rawsignal = y + (noise1 - noise2)

fig, ax = plt.subplots(figsize=(7,5))
ax.plot(x,rawsignal, label='signal + noise', color='cornflowerblue')
ax.plot(x,y, label='Original Gaussian signal', color='crimson')
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'signal + noise')
plt.legend()
plt.show()
```

It is clear that the Gaussian profile is completely hidden into the noise.

### Application of the Statistical Concepts. Extracting a Signal from the Noise.

Now we will use some statistic concepts such as mean and median to isolate the signal from the noise. First, let us create 9 of such signal + noise synthetic profiles.

```python
n = 9 # Number of profiles
rawprofiles = np.zeros([n,200])
for i in range(n):
  rawprofiles[i] = y + ( np.random.rand(200) - np.random.rand(200) )

fig, ax = plt.subplots(3,3, figsize=(10,7))

ax[0,0].plot(x,rawprofiles[0])
ax[0,1].plot(x,rawprofiles[1])
ax[0,2].plot(x,rawprofiles[2])
ax[1,0].plot(x,rawprofiles[3])
ax[1,1].plot(x,rawprofiles[4])
ax[1,2].plot(x,rawprofiles[5])
ax[2,0].plot(x,rawprofiles[6])
ax[2,1].plot(x,rawprofiles[7])
ax[2,2].plot(x,rawprofiles[8])

ax[2,1].set_xlabel(r'$x$')
ax[1,0].set_ylabel(r'$signal$')
plt.show()
```

The question here is ¿How can we recover the original signal from these profiles?

**Stacking** the profiles (signal+noise) will provide a method to recover the signal.

#### Stacking using the Mean

Since the profiles have a random noise, when adding regions with only noise, the mean of the random numbers cancel out. On the other hand, when we add regions in which there is some signal data, the mean adds together, increasing the so-called **signal to noise** ratio.

```python
recovered_signal = np.mean(rawprofiles, axis=0)

fig, ax = plt.subplots(figsize=(7,5))
ax.plot(x,recovered_signal, label='Recovered signal', color='cornflowerblue')
ax.plot(x,y, label='Original Gaussian signal', color='crimson')
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'signal')
plt.legend()
plt.show()
```

It is clear that the recovered signal is not perfect, although the Gaussian profile seems to be present.

Taking not 9 but 100 synthetic profiles, the stacking method gives a much better result for the recovered signal.

```python
n = 100
rawprofiles = np.zeros([n,200])
for i in range(n):
  rawprofiles[i] = y + ( np.random.rand(200) - np.random.rand(200) )

recovered_signal = np.mean(rawprofiles, axis=0)

fig, ax = plt.subplots(figsize=(7,5))
ax.plot(x,recovered_signal, label='Recovered signal', color='cornflowerblue')
ax.plot(x,y, label='Original Gaussian signal', color='crimson')
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'signal')
plt.legend()
plt.show()
```

Now, the Gaussian original signal is evident!

#### Stacking using the Median

Another form to calcualte the stack is using the median instead of the mean. When a distribution is symmetric the mean and the median are equivalent. However, if the distribution is asymmetric, or when there are significan outliers, the median can be a much better indicator of the central value.

Hence, although our random noise is expected to be a symmetric distribution around the zero  value, there may exist outliers that affect the result of the mean. Therefore we will make a median stacking of 9 signal+noise profiles to compare the results.

```python
n = 9
rawprofiles = np.zeros([n,200])
for i in range(n):
  rawprofiles[i] = y + ( np.random.rand(200) - np.random.rand(200) )

recovered_signal = np.median(rawprofiles, axis=0)

fig, ax = plt.subplots(figsize=(7,5))
ax.plot(x,recovered_signal, label='Recovered signal (Median Stacking)', color='cornflowerblue')
ax.plot(x,y, label='Original Gaussian signal', color='crimson')
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'signal')
plt.legend()
plt.show()
```

Note that the result is almost the same as that obtained with the mean stacking (the reason may be that the noise distribution is symmetric w.r.t. zero). Now we will consider 100 profiles,

```python
n = 100
rawprofiles = np.zeros([n,200])
for i in range(n):
  rawprofiles[i] = y + ( np.random.rand(200) - np.random.rand(200) )

recovered_signal = np.median(rawprofiles, axis=0)

fig, ax = plt.subplots(figsize=(7,5))
ax.plot(x,recovered_signal, label='Recovered signal (Median Stacking)', color='cornflowerblue')
ax.plot(x,y, label='Original Gaussian signal', color='crimson')
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'signal')
plt.legend()
plt.show()
```

Once again, the result is not much different from that obtained using the mean stacking.


# the German tank problem

the Germans have a population of $n$ tanks labeled with serial numbers $1,2,...,n$ on the tanks. The number of tanks $n$ is unknown and of interest to the Allied forces. The Allied forces randomly capture $k$ tanks from the Germans with replacement and observe their serial numbers $\{x_1, x_2, ..., x_k\}$. The goal is to, from observing the serial numbers on this random sample of the tanks, estimate $n$.

the *estimator* of $n$ maps an outcome of the experiment to an estimate of $n$, $\hat{n}$.

```julia
using StatsBase
using PyPlot
using Statistics
using Printf

PyPlot.matplotlib.style.use("seaborn-pastel")

rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16;
```

    ┌ Info: Recompiling stale cache file /home/mick/.julia/compiled/v1.2/StatsBase/EZjIG.ji for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
    └ @ Base loading.jl:1240

## data structure for a tank

for elegance

```julia
struct Tank
    serial_no::Int
end
tank = Tank(3)
```

    Tank(3)

## visualizing the captured tanks and their serial numbers

```julia
function viz_tanks(tanks::Array{Tank}, savename=nothing)
    nb_tanks = length(tanks)

    img = PyPlot.matplotlib.image.imread("tank.png")

    fig, ax = subplots(1, nb_tanks)
    for (t, tank) in enumerate(tanks)
        ax[t].imshow(img)
        ax[t].set_title(tank.serial_no)
        ax[t].axis("off")
    end
    tight_layout()

    if ! isnothing(savename)
        savefig(savename * ".png", format="png", dpi=300)
        # Linux command line tool to trim white space
        run(`convert $savename.png -trim $savename.png`)
    end
end

n = 7
tanks = [Tank(s) for s in 1:n]
viz_tanks(tanks)
```

## simulating tank capture

write a function `capture_tanks` to simulate the random sampling of `nb_tanks_captured` tanks from all `nb_tanks` tanks the Germans have (without replacement). return a random sample of tanks.

```julia
function capture_tanks(num_captured::Int, num_tanks::Int)
    return sample([Tank(i) for i in 1:num_tanks], num_captured, replace=false)
end

viz_tanks(capture_tanks(3, 100))
```

## defining different estimators

an estimator maps an outcome $\{x_1, x_2, ..., x_k\}$ to an estimate for $n$, $\hat{n}$.

### estimator (1): maximum serial number

this is the maximum likelihood estimator.

\begin{equation}
\hat{n} = \max_i x_i
\end{equation}

```julia
function max_serial_no(captured_tanks::Array{Tank})
    return maximum([t.serial_no for t in captured_tanks])
end

max_serial_no(tanks)
```

    7

### estimator (2): maximum serial number plus initial gap

\begin{equation}
\hat{n} = \max_i x_i + \bigl(\min_i x_i -1\bigr)
\end{equation}

```julia
function max_plus_initial_gap(captured_tanks::Array{Tank})
    gap_adjustment = minimum([t.serial_no for t in captured_tanks]) - 1
    return max_serial_no(captured_tanks) + gap_adjustment
end

max_plus_initial_gap(tanks)
```

    7

### estimator (3): maximum serial number plus gap if samples are evenly spaced

\begin{equation}
\hat{n} = \max_i x_i + \bigl( \max_i x_i / k -1 \bigr)
\end{equation}

```julia
function max_plus_even_gap(captured_tanks::Array{Tank})
    gap_adjustment = maximum([t.serial_no for t in captured_tanks]) / length(captured_tanks) - 1
    return max_serial_no(captured_tanks) + Int(gap_adjustment)
end

max_plus_even_gap(tanks)
```

    7

## assessing the bias and variance of different estimators

say the Germans have `nb_tanks` tanks, and we randomly capture `nb_tanks_captured`. what is the distribution of the estimators (over different outcomes of this random experiment), and how does the distribution compare to the true `nb_tanks`?

```julia

```

```julia

```

```julia

```

notes:

## what happens as we capture more and more tanks, i.e. increase $k$?

assess estimator (3).

```julia

```

## one-sided confidence interval

how confident are we that the Germans don't have *more* tanks?

significance level: $\alpha$

test statistic = estimator (3) = $\hat{n} = \max_i x_i + \bigl( \max_i x_i / k -1 \bigr)$

**null hypothesis**: the number of tanks is $n=n_0$<br>
**alternative hypothesis**: the number of tanks is less than $n_0$

we reject the null hypothesis (say, "the data does not support the null hypothesis") that the number of tanks is $n=n_0$ if the p-value is less than $\alpha$. the p-value is the probability that, if the null hypothesis is true, we get a test statistic equal to or smaller than we observed.

we want to find the highest $n_0$ such that we have statistical power to reject the null hypothesis in favor of the alternative hypothesis. this is the upper bound on the confidence interval!

then the idea is that, if the null hypothesis is that the number of tanks is absurdly large compared to the largest serial number we saw in our sample, it would be very unlikely that we would see such small serial numbers compared to the number of tanks, so we'd reject the null hypothesis.

```julia

```

say $\alpha=0.05$ and we seek a 95% one-sided confidence interval

```julia

```

```julia

```

In-Class Notes/German Tank Problem/.ipynb_checkpoints/German tank problem_sparse-checkpoint.ipynb


## Moving average filter

The moving average(MA) filter is a Low Pass FIR used for smoothing signals. This filter sum the data of L consecutive elements of the input vector and divide by L, therefore the result is a single output point.
As the parameter L increases, the smoothness of the output is better, whereas the sharp transitions in the data are made increasingly blunt. This implies that this filter has an excellent time-domain response but a poor frequency response.

### Implementation

\begin{align}
y[n]=\frac{1}{L}\sum_{k=0}^{L-1} x[n-k]
\end{align}

Where,

y: output vector

x: input vector

L: data point

Figure 1: Discrete-time 4-point Moving Average FIR filter

## Modules

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.signal import freqz, dimpulse, dstep
from math import sin, cos, sqrt, pi,pow
```

## Parameters

```python
fsampling = 100
L = 4
```

## Coefficients

```python
#coefficients
b = np.ones(L) #numerator coeffs of filter transfer function
#[1. 1. 1. 1.]
#b = (np.ones(L))/L #numerator coeffs of filter transfer function

a = np.array([L] + [0]*(L-1)) #denominator coeffs of filter transfer function
#[4 0 0 0]
#a = np.ones(1)  #denominator coeffs of filter transfer function
```

## Frequency response

```python
#frequency response
w, h = freqz(b, a, worN=4096)
#w, h = freqz(b, a)

w *= fsampling / (2 * pi)

# Plot the amplitude response
plt.figure(dpi=100)
plt.subplot(2, 1, 1)
plt.suptitle('Bode Plot')
plt.plot(w, 20 * np.log10(abs(h)))
plt.ylabel('Magnitude [dB]')
plt.xlim(0, fsampling / 2)
plt.ylim(-60, 10)
plt.axhline(-6.01, linewidth=0.8, color='black', linestyle=':')

# Plot the phase response
plt.subplot(2, 1, 2)
plt.plot(w, 180 * np.angle(h) / pi)
plt.xlabel('Frequency [Hz]')
plt.ylabel('Phase [°]')
plt.xlim(0, fsampling / 2)
plt.ylim(-180, 90)
plt.yticks([-180, -135, -90, -45, 0, 45, 90])
plt.show()
print("Figure 2: Magnitude and phase response of L=4-point Moving Average filter")
```

## Impulse response

```python
t, y = dimpulse((b, a, 1/fsampling), n=2*L)
plt.figure(dpi=100)
plt.suptitle('Impulse Response')
_, _, baseline = plt.stem(t, y[0], basefmt='k:')
plt.setp(baseline, 'linewidth', 1)
baseline.set_xdata([0,1])
baseline.set_transform(plt.gca().get_yaxis_transform())
plt.xlabel('Time [seconds]')
plt.ylabel('Output')
plt.xlim(-1/fsampling, 2*L/fsampling)
plt.yticks([0, 0.5/L, 1.0/L])
plt.show()
print("Figure 3: Plot the impulse response of discrete-time system.")
```

### Testing our equation

\begin{align}
|H(e^{j\omega})|=\sqrt{\frac{1}{16}((1+cos(\omega)+cos(2\omega)+cos(3\omega))^{2}+(sin(\omega)+sin(2\omega)+sin(3\omega))^{2})}
\end{align}

```python
N=4096 #Elements for w vector
w=(np.linspace(0, pi, N, endpoint=True)).reshape(N, )

H=np.zeros((N,1))
for i in range(N-2):
    H[i]=sqrt(pow(1/L,2)*(pow(1+cos(w[i])+cos(2*w[i])+cos(3*w[i]),2)+ pow(sin(w[i])+sin(2*w[i])+sin(3*w[i]),2)))

```

#### Comparison of results

```python
plt.figure(dpi=100)
plt.plot(w*fsampling / (2 * pi),20*np.log10(H+0.000000001))
plt.plot(w*fsampling / (2 * pi), 20*np.log10(abs(h)),linestyle='dashed')

plt.ylabel('Magnitude [dB]')
plt.xlabel('Frequency [Hz]')

plt.xlim(0, fsampling / 2)
plt.ylim(-60, 10)
plt.axhline(-6.01, linewidth=0.8, color='black', linestyle=':')
plt.show()
print("Figure 4: Comparison of freqz vs our equation")
```

## Resources

SciPy Documentation: scipy.signal.freqz

```python

```


# Taylor integration example: $\dot x=x^2$

Here, we will integrate the initial-value problem (IVP) defined by

$$
\begin{align}
\dot x &= x^2 \\
x(0)&=x_0
\end{align}
$$

Given a real number $A>0$, the restriction of the function $f(x)=x^2$ over the interval $I_A=(-A,A)$ satisfies the Lipschitz condition $|f(x_1)-f(x_2)|=|x_1^2-x_2^2|=|x_1+x_2|\cdot|x_1-x_2| \leq 2A|x_1-x_2|$ for every $x_1,x_2 \in I_A$. Therefore, the Picard-Lindelöf theorem for ordinary differential equations guarantees there exists a $\delta>0$ such that a solution for this IVP exists and is unique for $t \in [0,\delta]$, for any $x_0 \in I_A$. Note that in this case, it is necessary to restrict the function to a bounded interval in order to obtain a Lipschitz condition, which in turn is necessary to fulfill the conditions of the Picard-Lindelöf theorem.

Now, for any initial condition $x_0$, $t_0\in\mathbb{R}$ the analytical solution for this problem is:

$$
x(t)=\frac{x_0}{1-x_0\cdot (t-t_0)}
$$

In particular, the analytical solution exhibits a divergence at $t^*=t_0+1/x_0$; i.e., the solution is guaranteed to exist only for $t \in [t_0,t_0+1/x_0)$ if $x_0>0$. Otherwise, if $x_0<0$, then the analytical solution "moves away" from the singularity, since in this case we have $t^*=t_0+1/x_0<t_0$ (i.e., if $x_0<0$, the singularity is in the "past"). How does a numerical integrator behave near this divergence, when $x_0>0$?

We will try three methods to integrate this problem:

+ Adaptive time-step, 4th-order, Runge-Kutta (`ODE.jl`)
+ Taylor method (`TaylorIntegration.jl`)
+ Adaptive time-step, Runge-Kutta-Fehlberg 7/8 method (`ODE.jl`)

As initial conditions to integrate this IVP, we choose $x_0=3$, $t_0=0$, since then the singularity will be at $t^*=1/3$, and this number is not exactly representable in a binary floating-point format. Thus, any constant time-step numerical integrator should break down when integrating up to $t_\mathrm{max}=1/3$, or beyond.

We start off by including the relevant packages:

```julia
using TaylorIntegration, ODE, Plots, LaTeXStrings
```

The ODE:

```julia
diffeq(t, x) = x.^2
```

    diffeq (generic function with 1 method)

## 1. Adaptive time-step, 4th order, Runge Kutta method

We select $x_0=3$, $t_0=0$. Then, the singularity is at $t=1/3$.

```julia
@time tRK, xRK = ode45(diffeq, 3.0, [0.0, 0.34]); #warmup lap
@time tRK, xRK = ode45(diffeq, 3.0, [0.0, 0.34]);
```

    Warning: dt < minstep.  Stopping.
      1.397936 seconds (1.51 M allocations: 64.582 MB, 1.80% gc time)
    Warning: dt < minstep.  Stopping.
      0.000970 seconds (11.54 k allocations: 465.828 KB)

Plot $x$ vs $t$ (log-log):

```julia
pyplot()
plot(log10(tRK[2:end]), log10(xRK[2:end]))
title!("x vs t (log-log)")
xlabel!(L"\log_{10}(t)")
ylabel!(L"\log_{10}(x(t))")
```

    sys:1: MatplotlibDeprecationWarning: The set_axis_bgcolor function was deprecated in version 2.0. Use set_facecolor instead.

What is the final state of the system?

```julia
tRK[end], xRK[end]
```

    (0.33333423758781194,7.125446356124256e17)

Does the integrator get past the singularity?

```julia
tRK[end]>1/3
```

true

The answer is yes! So the last value of the solution is meaningless:

```julia
xRK[end] #this value is meaningless
```

7.125446356124256e17

How many steps did the RK integrator perform?

```julia
length(xRK)-1
```

166

How does the numerical solution compare to the analytical solution? The analytical solution is:

```julia
exactsol(t, x0) = x0./(1.0-x0.*t) #analytical solution
```

    exactsol (generic function with 1 method)

The relative difference between the numerical and analytical solution, $\delta x$, is:

```julia
δxRK = (xRK-exactsol(tRK, 3.0))./exactsol(tRK, 3.0) #numerical error, relative to analytical solution
;
```

The $\delta x$ vs $t$ plot (semilog):

```julia
plot(tRK[2:end], log10(abs(δxRK[2:end])))
title!("Relative error (semi-log)")
xlabel!(L"\log_{10}(t)")
ylabel!(L"\log_{10}(\delta x(t))")
```

This plot means that the error of the numerical solution grows systematically; and at the end of the integration, the error in the numerical solution is

```julia
(xRK[end-1]-exactsol(tRK[end-1], 3.0))
```

5.433622064235371e17

## 2. Taylor method

Again, we select $x_0=3$, $t_0=0$. The order of the Taylor integration is $28$, and we set the absolute tolerance equal to $10^{-20}$; this value is used during each time-step in order to compute an adaptive step size. We set the maximum number of integration steps equal to the number of steps that the previous integrator did.

```julia
@time tT, xT = taylorinteg(diffeq, 3.0, 0.0, 0.34, 28, 1e-20, maxsteps=length(xRK)-1); #warmup lap
@time tT, xT = taylorinteg(diffeq, 3.0, 0.0, 0.34, 28, 1e-20, maxsteps=length(xRK)-1);
```

    WARNING: Maximum number of integration steps reached; exiting.

      0.215803 seconds (160.86 k allocations: 8.295 MB)
      0.001468 seconds (19.65 k allocations: 2.171 MB)

```julia
tT[end], xT[end]
```

    (0.3333333329479479,2.5948055925168757e9)

How many steps did the Taylor integrator perform?

```julia
length(xT)-1
```

166

Below, we show the $x$ vs $t$ plot (log-log):

```julia
plot(log10(tT[2:end]), log10(xT[2:end]))
title!("x vs t (log-log)")
xlabel!(L"\log_{10}(t)")
ylabel!(L"\log_{10}(x(t))")
```

Does the integrator get past the singularity?

```julia
tT[end] > 1/3
```

false

The answer is no! Even if we increase the value of the `maxsteps` keyword in `taylorinteg`, it doesn't get past the singularity!

Now, the relative difference between the numerical and analytical solution, $\delta x$, is:

```julia
δxT = (xT.-exactsol(tT, 3.0))./exactsol(tT, 3.0);
```

The $\delta x$ vs $t$ plot (logscale):

```julia
plot(tT[6:end], log10(abs(δxT[6:end])))
title!("Relative error (semi-log)")
xlabel!(L"t")
ylabel!(L"\log_{10}(\delta x(t))")
```

We observe that, while the execution time is ~10 times longer wrt 4th-order RK, the numerical solution obtained by the Taylor integrator stays within $10^{-12}$ of the analytical solution, for a same number of steps.

Now, that happens if we use a higher order Runge Kutta method to integrate this problem?

## 3. Runge-Kutta-Fehlberg 7/8 method

Here we use the Runge-Kutta-Fehlberg 7/8 method, included in `ODE.jl`, to integrate the same problem as before.

```julia
@time t78, x78 = ode78(diffeq, 3.0, [0.0, 0.34]); #warmup lap
@time t78, x78 = ode78(diffeq, 3.0, [0.0, 0.34]);
```

    Warning: dt < minstep.  Stopping.
      0.379769 seconds (345.22 k allocations: 15.063 MB, 2.40% gc time)
    Warning: dt < minstep.  Stopping.
      0.001199 seconds (12.76 k allocations: 494.219 KB)

Plot $x$ vs $t$ (log-log):

```julia
plot(log10(t78[2:end]), log10(x78[2:end]))
title!("x vs t (log-log)")
xlabel!(L"\log_{10}(t)")
ylabel!(L"\log_{10}(x(t))")
```

What is the final state of the system?

```julia
t78[end], x78[end]
```

    (0.3333336190078445,1.6711546943686948e18)

Does the integrator get past the singularity?

```julia
t78[end]>1/3
```

true

The answer is yes! So the last value of the solution is meaningless:

```julia
x78[end] #this value is meaningless
```

1.6711546943686948e18

How many steps did the RK integrator perform?

```julia
length(x78)-1
```

91

The relative difference between the numerical and analytical solution, $\delta x$, is:

```julia
δx78 = (x78-exactsol(t78, 3.0))./exactsol(t78, 3.0) #error relative to analytical solution
;
```

The $\delta x$ vs $t$ plot (semilog):

```julia
plot(t78[2:end], log10(abs(δx78[2:end])))
title!("Relative error (semi-log)")
xlabel!(L"t")
ylabel!(L"\log_{10}(\delta x(t))")
```

This time, the RKF 7/8 integrator is "only" twice as fast as the Taylor integrator, but the error continues to be greater than the error from the latter by several orders of magnitude.

## 4. Adaptive 4th-order RK, stringer tolerance

As a last example, we will integrate once again our problem using a 4th-order adaptive RK integrator, but imposing a stringer tolerance:

```julia
@time tRK_, xRK_ = ode45(diffeq, 3.0, [0.0, 0.34], abstol=1e-8, reltol=1e-8 ); #warmup lap
@time tRK_, xRK_ = ode45(diffeq, 3.0, [0.0, 0.34], abstol=1e-8, reltol=1e-8 );
;
```

    Warning: dt < minstep.  Stopping.
      0.127591 seconds (383.26 k allocations: 16.016 MB, 6.19% gc time)
    Warning: dt < minstep.  Stopping.
      0.029600 seconds (335.10 k allocations: 13.837 MB, 31.90% gc time)

Now, the integrator takes 10 times longer to complete the integration than the Taylor method.

Does it get past the singularity?

```julia
tRK_[end] > 1/3
```

true

Yes! So, once again, the last value reported by the integrator is completely meaningless. But, has it attained a higher precision than the Taylor method? Well, let's calculate once again the numerical error relative to the analytical solution:

```julia
δxRK_ = (xRK_-exactsol(tRK_, 3.0))./exactsol(tRK_, 3.0);
```

And now, let's plot this relative error vs time:

```julia
plot(tRK_[2:end], log10(abs(δxRK_[2:end])))
title!("Relative error (semi-log)")
xlabel!(L"t")
ylabel!(L"\log_{10}(\delta x(t))")
ylims!(-20,20)
```

The numerical error has actually gotten worse! `TaylorIntegration.jl` is indeed a really competitive package to integrate ODEs.

```julia

```


# Nonlinear Equations and bisection

A large number of  nonlinear equations obviously *have* solutions, but the solutions cannot always be found in closed form. For example,

\begin{equation}
  f(x) = e^x + x - 2 = 0.
\end{equation}

```
%matplotlib inline
```

```
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
rcParams['figure.figsize'] = (12,6)
```

```
x = np.linspace(0.0, 1.0)
def fn(x):
    return np.exp(x) + x - 2.0
f = fn(x)

plt.figure()
plt.plot(x, f, label = r'$f(x) = e^x + x - 2$')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.legend(loc = 2)
plt.show()
```

Numerical methods can give accurate, approximate solutions.

Write the problem as

\begin{equation}
    f(x) = 0.
\end{equation}

Goal: given $f$, find $x$.

Assume: $f$ real, continuous. Sometimes restrict to $f$ differentiable.

May consider equivalent problem

\begin{equation}
    g(x) = x.
\end{equation}

Same problem, different (geometric) interpretation.

Assume that evaluating $f$ or $g$ is \emph{expensive}. Want to minimize the number of evaluations.

Given $f: x \in \mathbb{R} \to \mathbb{R}$, continuous, find $x$ such that

\begin{equation}
    f(x) = 0.
\end{equation}

Simple and robust method: **bisection**.

1. Assume we have found (somehow!) two points, $x^{(L)}$ and $x^{(R)}$ that bracket root.
2. Check halfway point $x^{(M)}$. Either $f(x^{(M)})=0$ (problem solved), or $x^{(M)}$ and one of $x^{(L,R)}$ bracket the root.
3. Repeat to converge.

```
xL = 0.1
xR = 0.9
xM = 0.5 * (xL + xR)

plt.figure()
plt.plot(x, f)
plt.plot([xL], [fn(xL)], 'bo'); plt.plot([xL, xL], [0, fn(xL)], 'b--')
plt.plot([xR], [fn(xR)], 'bo'); plt.plot([xR, xR], [0, fn(xR)], 'b--')
plt.plot([xM], [fn(xM)], 'go'); plt.plot([xM, xM], [0, fn(xM)], 'g--')
plt.axhline(color = 'black', linestyle = '--')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.show()
```

Lectures/05 - Nonlinear equations and bisection.ipynb


# Mathematical Theory and Intuition behind our improved model

This markdown explains the mathematical theory behind our model. Note that the actual implementation might be different. For simplicity, we are also not including all the datasets we used (otherwise the equations might be too long).

Let $i$ denote the $i$th date since the beginning of our dataset and $n$ denote the cutoff date for training-testing . Let the range from $n+1$ to $m$ be the range for testing (forecast).

For a given date $i$, let $\textbf{asp}(i)$ be the actual index for S\&P500; $\textbf{fbsp}(i)$ be the index for S\&P500 bases on a training set of the first $i-1$ many days' market data of S\&P500; $\textbf{div}(i)$ be the dividend yield rates of S\&P500; $\textbf{eps}(i)$ be the dividen yield rates of S\&P500; $\textbf{tby}(i)$ be the 10-year treasury bond yield; $\textbf{ffr}(i)$ be the fed fund rates;  $\textbf{fta}(i)$ be the fed total assets.

## Linear Model

\begin{equation}
    \textbf{fb}_1(i,a,b,c,d,e):=\textbf{fbsp}(i)+a*\textbf{tby}(i)+b*\textbf{ffr}(i)+c*\textbf{fta}(i)+d*\textbf{div}(i)+e*\textbf{eps}(i)
\end{equation}

\begin{equation}
    \textbf{E}_n(a,b,c,d,e):= \frac{1}{n-1000}\sum_{i=1000}^{n}
    (\textbf{fb}_1(i,a,b,c,d,e)-\textbf{asp}(i))^2
\end{equation}

attains its minimum. Namely, finding

\begin{equation}
    (a_n,b_n,c_n,d_n,e_n):=\text{argmin} \textbf{E}_n(a,b,c,d,e)
\end{equation}

Note that it doesn't make sense to start with $i=1$ since the fbprophet itself need to use the first $i-1$ of the data for training

## Nonlinear Model

All notations are the same as above.

Here is a slightly different model (nonlinear revision of fbprophet). In this model, we will the dividend yield rate of S\&P 500.

First, what is the dividend yield rate? Dividend is the money that publicly listed companies pay you (usually four times a year) for holding their stocks (until the so-called ex-dividend dates). It's like the interest paid to your saving accounts from the banks. Some companies pay while some don't, especially for the growth tech stocks. From my experience, the impact of bond rates and fed fund rates on the stock market will change when they rise above or fall below the dividend yield rate}. Stock prices fall when those rates rise above the dividend yield rate of SP500 (investor are selling their stocks to buy bonds or save more money in their bank account!).

Based on this idea, it might be useful to consider the differences of those rates and the dividend yield rate of SP500.

Normally an increase in the federal fund rate will result in an increase in bank loan interest rate, which will in turn result in an decrease in the net income of S\&P500-listed companies since they have to pay a higher interest when borrowing money from banks. Based on this thought, I believe it is reasonable to make a correction to $c*\textbf{eps}(i)$ by replacing the term by $c*\textbf{eps}(i)(1+d*\textbf{ffr}(i))$. If my intuition is correct, the generated constant $d$ from the optimization should be a negative number.

$$\textbf{fb}_2(i,a,b,c,d,e):= \textbf{fbsp}(i)*\big[1+a*(\textbf{div}(i)-\textbf{tby}(i))\big]\big[1+b*(\textbf{div}(i)-\textbf{ffr}(i))\big]+c*\textbf{eps}(i)(1+d*\textbf{ffr}(i)+e*\textbf{fta}(i))$$

and consider

$$E_n(a,b,c,d,e) := \frac{1}{n-1000}\sum_{i=1000}^n(\text{fb}_2(i,a,b,c,d,e)-\textbf{asp}(i))^2$$

Now find (by approximation, SGD, etc.) $(a_n,b_n,c_n,d_n,e_n):=\text{argmin} E_n(a,b,c,d,e)$

Using $(a_n,b_n,c_n,d_n,e_n)$ as constants, our output will be $\textbf{fb}_2(i,a_n,b_n,c_n,d_n,e_n)$

### The actual implementation of the nonlinear model

For the actual implementation of the nonlinear model, we threw away the higher order terms (the products of three things) since they are relatively smaller quantities

## A faster computation scheme
Here is a method we could think of that may reduce the number of times calling fbprophet:

Say in the training process we are considering from i=1,000 to 11,000. Namely based on the current scheme, we have to call fbprophet for 10,000 times.

Instead of this, we make a break-down 10,000=100*100 as follows:

For i=1,000 to 1,100, we only use the first 999 dates for training;

For i=1,100 to 1,200, we only use the first 1,099 dates for training;

.............................................

For i=10,900 to 11,000, we only use the first 10,899 dates for training;

In this way, it seems that we only need to call fbprophet for 100 times. And this doesn't seem harm the accuracy too much.

```python
import pandas as pd
import numpy as np

## For plotting
import matplotlib.pyplot as plt
from matplotlib import style
import datetime as dt
import seaborn as sns
sns.set_style("whitegrid")
```

```python
path = '../Data/dff1.csv'
```

```python
df= pd.read_csv(path, parse_dates=['ds'])
# df = df.rename(columns = {"Date":"ds","Close":"y"})
df = df[['ds', 'y','fbsp', 'diff','tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi',
       'rfs']]
# df
```

```python
df['fbsp_tby'] = df['fbsp'] * df['tby']
df['fbsp_ffr'] = df['fbsp'] * df['ffr']
df['fbsp_div'] = df['fbsp'] * df['div']
df['eps_tby'] = df['eps'] * df['tby']
df['eps_ffr'] = df['eps'] * df['ffr']
df['eps_div'] = df['eps'] * df['div']
```

```python
# cutoff between test and train data
cutoff = len(df) - 252
df_train = df[:cutoff].copy()
df_test = df[cutoff:].copy()
print(cutoff)
```

    2300

```python
df_train.columns
```

    Index(['ds', 'y', 'fbsp', 'diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une',
           'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
           'eps_ffr', 'eps_div'],
          dtype='object')

```python
possible_features = ['tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti',
       'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
       'eps_ffr', 'eps_div']

from itertools import chain, combinations

def powerset(iterable):
    #"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

#print(list(powerset(possible_features)))
```

```python
len(possible_features)
```

    15

```python
from statsmodels.regression.linear_model import OLS

reg_new = OLS((df_train['diff']).copy(),df_train[possible_features].copy()).fit()
print(reg_new.params)

#from the output, we can see it's consistent with sklearn output
```

    tby         305.980303
    ffr         183.608289
    fta           0.000065
    eps          -1.775823
    div        -142.530009
    une         -80.692625
    wti           3.378638
    ppi          -7.923858
    rfs           0.006087
    fbsp_tby     -0.043153
    fbsp_ffr     -0.146870
    fbsp_div     -0.391168
    eps_tby      -2.122760
    eps_ffr       2.390346
    eps_div       3.950103
    dtype: float64

```python
new_coef = reg_new.params
new_possible_feats = new_coef[abs(new_coef)>0].index

power_feats = list(powerset(new_possible_feats))
power_feats.remove(())

power_feats = [ list(feats) for feats in power_feats]
len(power_feats)

```

    32767

```python
AIC_scores = []
parameters = []

for feats in power_feats:
    tmp_reg = OLS((df_train['diff']).copy(),df_train[feats].copy()).fit()
    AIC_scores.append(tmp_reg.aic)
    parameters.append(tmp_reg.params)

Min_AIC_index = AIC_scores.index(min(AIC_scores))
Min_AIC_feats = power_feats[Min_AIC_index]
Min_AIC_params  = parameters[Min_AIC_index]
print(Min_AIC_feats)
print(Min_AIC_params)
```

    ['tby', 'ffr', 'fta', 'div', 'une', 'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']
    tby         313.791358
    ffr         187.937377
    fta           0.000057
    div         -78.607290
    une         -87.125044
    wti           3.455364
    ppi          -8.104355
    rfs           0.005916
    fbsp_tby     -0.046476
    fbsp_ffr     -0.132070
    fbsp_div     -0.388510
    eps_tby      -2.157124
    eps_ffr       2.033978
    eps_div       3.252783
    dtype: float64

```python
len(Min_AIC_feats)
```

    14

```python
###After selecting the best features, we report the testing error, and make the plot
AIC_df_test = df_test[Min_AIC_feats]
AIC_pred_test = AIC_df_test.dot(Min_AIC_params)+df_test.fbsp

AIC_df_train = df_train[Min_AIC_feats]
AIC_pred_train = AIC_df_train.dot(Min_AIC_params)+ df_train.fbsp

```

```python
from sklearn.metrics import mean_squared_error as MSE

mse_train = MSE(df_train.y, AIC_pred_train)
mse_test = MSE(df_test.y, AIC_pred_test)

#compare with fbprophet()

fb_mse_train = MSE(df_train.y, df_train.fbsp)
fb_mse_test = MSE(df_test.y, df_test.fbsp)

print(mse_train,fb_mse_train)

print(mse_test,fb_mse_test)
```

    2543.6188296247346 22303.56360854362
    11928.655059590139 15247.912341091065

```python
df_train.ds
```

    0      2009-12-15
    1      2009-12-16
    2      2009-12-17
    3      2009-12-18
    4      2009-12-21
              ...
    2295   2019-02-20
    2296   2019-02-21
    2297   2019-02-22
    2298   2019-02-25
    2299   2019-02-26
    Name: ds, Length: 2300, dtype: datetime64[ns]

```python
plt.figure(figsize=(18,10))

# plot the training data
plt.plot(df_train.ds,df_train.y,'b',
            label = "Training Data")

plt.plot(df_train.ds, AIC_pred_train,'r-',
            label = "Improved Fitted Values by Best_AIC")

# # plot the fit
plt.plot(df_train.ds, df_train.fbsp,'g-',
            label = "FB Fitted Values")

# # plot the forecast
plt.plot(df_test.ds, df_test.fbsp,'g--',
            label = "FB Forecast")
plt.plot(df_test.ds, AIC_pred_test,'r--',
            label = "Improved Forecast by Best_AIC")
plt.plot(df_test.ds,df_test.y,'b--',
            label = "Test Data")

plt.legend(fontsize=14)

plt.xlabel("Date", fontsize=16)
plt.ylabel("SP&500 Close Price", fontsize=16)

plt.show()
```

```python
plt.figure(figsize=(18,10))
plt.plot(df_test.y,label="Training Data")
plt.plot(df_test.fbsp,label="FB Forecast")
plt.plot(AIC_pred_test,label="Improved Forecast by Best_AIC")
plt.legend(fontsize = 14)
plt.show()
```

```python

```

```python
column = ['tby', 'ffr', 'fta', 'eps', 'div', 'une',
                                                  'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']
```

```python
from sklearn import preprocessing
df1_train = df_train[['diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']]

X = preprocessing.scale(df1_train)
from statsmodels.regression.linear_model import OLS

reg_new = OLS((X[:,0]).copy(),X[:,1:].copy()).fit()
print(reg_new.params)
```

    [ 1.50405129  1.03228322  0.27409454  1.17073571  0.31243092 -0.75747342
      0.46988206 -0.39944639  2.10369448 -0.69112943 -2.1804296  -2.38576385
     -1.14196633  1.41832903 -0.34501927]

```python
# Before Covid
# pd.Series(reg_new.params, index=['tby', 'ffr', 'fta', 'eps', 'div', 'une',
#                                                   'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'] )
```

```python
# before covid
coef1 = [ 1.50405129,  1.03228322,  0.27409454,  1.17073571,  0.31243092,
       -0.75747342,  0.46988206, -0.39944639,  2.10369448, -0.69112943,
       -2.1804296 , -2.38576385, -1.14196633,  1.41832903, -0.34501927]
# include covid
coef2 = [ 0.65150054,  1.70457239, -0.1573802 , -0.18007979, -0.15221931,
       -0.62326075,  0.45065894, -0.38972706,  2.87210843, -1.17604495,
       -4.92858316, -2.15459111,  0.11418468,  2.74829778,  0.55520382]
```

```python
# Include Covid
# pd.Series( np.append( ['coefficients (before covid)'], np.round(coef1,3)),  index= np.append(['features'], column) )

```

```python
index1 = ['10 Year U.S Treasury Bond Yield Rates (tby)', 'Federal Funds Rates (ffr)',
        'Federal Total Assets (fta)', 'Earning-Per-Share of S&P 500 (eps)', 'Dividend Yield of S&P 500 (div)',
        'Unemployment Rates (une) ', 'West Texas Intermediate oil index (wit)', 'Producer Price Index (ppi)',
         'Retail and Food Services Sales (rfs)',
         'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'
        ]
```

```python
len(index1)
```

    15

```python
pd.Series(coef2, index =index1)
```

    10 Year U.S Treasury Bond Yield Rates (tby)    0.651501
    Federal Funds Rates (ffr)                      1.704572
    Federal Total Assets (fta)                    -0.157380
    Earning-Per-Share of S&P 500 (eps)            -0.180080
    Dividend Yield of S&P 500 (div)               -0.152219
    Unemployment Rates (une)                      -0.623261
    West Texas Intermediate oil index (wit)        0.450659
    Producer Price Index (ppi)                    -0.389727
    Retail and Food Services Sales (rfs)           2.872108
    fbsp_tby                                      -1.176045
    fbsp_ffr                                      -4.928583
    fbsp_div                                      -2.154591
    eps_tby                                        0.114185
    eps_ffr                                        2.748298
    eps_div                                        0.555204
    dtype: float64

```python
df3 = pd.DataFrame(coef1, index = index1, columns = ['coefficients (before covid)'])
df3['coefficients (include covid)'] =pd.Series(coef2, index =index1)
df3
```

      coefficients (before covid)
      coefficients (include covid)

      10 Year U.S Treasury Bond Yield Rates (tby)
      1.504051
      0.651501

      Federal Funds Rates (ffr)
      1.032283
      1.704572

      Federal Total Assets (fta)
      0.274095
      -0.157380

      Earning-Per-Share of S&P 500 (eps)
      1.170736
      -0.180080

      Dividend Yield of S&P 500 (div)
      0.312431
      -0.152219

      Unemployment Rates (une)
      -0.757473
      -0.623261

      West Texas Intermediate oil index (wit)
      0.469882
      0.450659

      Producer Price Index (ppi)
      -0.399446
      -0.389727

      Retail and Food Services Sales (rfs)
      2.103694
      2.872108

      fbsp_tby
      -0.691129
      -1.176045

      fbsp_ffr
      -2.180430
      -4.928583

      fbsp_div
      -2.385764
      -2.154591

      eps_tby
      -1.141966
      0.114185

      eps_ffr
      1.418329
      2.748298

      eps_div
      -0.345019
      0.555204

```python

```

Code/.ipynb_checkpoints/Scenario2 excluding the covid period-checkpoint.ipynb


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Polynomial Parametric Curves

```python
%matplotlib inline
```

```python
import sympy as sp
sp.init_printing(order='grevlex')
```

```python
t = sp.symbols('t')
```

The coefficients are written as bold letters, because they are elements of a vector space (e.g. $\mathbb{R}^3$).

```python
coefficients = sp.Matrix(sp.symbols('abm:4')[::-1])
coefficients
```

Monomial basis functions:

```python
b_monomial = sp.Matrix([t**3, t**2, t, 1]).T
b_monomial
```

```python
b_monomial.dot(coefficients)
```

This is a cubic polynomial in its canonical form (monomial basis).

Monomial basis functions:

```python
sp.plot(*b_monomial, (t, 0, 1));
```

It doesn't look like much, but every conceivable cubic polynomial can be formulated as exactly one linear combination of those basis functions.

Example:

```python
example_polynomial = (2 * t - 1)**3 + (t + 1)**2 - 6 * t + 1
example_polynomial
```

```python
sp.plot(example_polynomial, (t, 0, 1));
```

Can be re-written with monomial basis functions:

```python
example_polynomial.expand()
```

```python

```


# Linear Algebra

### Vector Space

A ***vector space*** $V$ defined on the field $K$ is ...

It satisfies the 10 following properties:
1.

A ***vector*** is an element of a vector space.

### Linear Transformation

Mapping between 2 vector spaces.

### Null and range space

Assume two vector spaces $V$ and $W$, and a linear mapping $\pmb{A}: V \rightarrow W$.

* The null (aka kernel) space of a matrix $\pmb{A}$ is defined as the subspace of $V$ such that:

\begin{equation}
    \mathcal{N}(\pmb{A}) = \{\pmb{x} \in V : \pmb{Ax} = \pmb{0}\} \subseteq V
\end{equation}

* The range (aka column or image) space of a matrix $\pmb{A}$ is defined as the subspace of $W$ such that:

\begin{equation}
    \mathcal{R}(\pmb{A}) = \{\pmb{y} \in W : \pmb{Ax} = \pmb{y}, \; \forall \pmb{x} \in V \} \subseteq W
\end{equation}

By the ***Rank–nullity theorem***:
\begin{equation}
    dim(\mathcal{N}(\pmb{A})) + dim(\mathcal{R}(\pmb{A})) = dim(V)
\end{equation}

### Rank of a matrix

### Transpose

### Determinant

Determinant of a square matrix.

\begin{equation}
    det : \mathbb{K}^{N \times N} \rightarrow \mathbb{K}: det(\pmb{A}) = |\pmb{A}|
\end{equation}

Properties:
* $det(\pmb{A}) = det(\pmb{A}^T)$
* $det(\pmb{A}^{-1}) = det(\pmb{A})^{-1}$
* If $\pmb{A}$ and $\pmb{B}$ are square matrices of same size then $det(\pmb{A}\pmb{B}) = det(\pmb{A}) det(\pmb{B}) = det(\pmb{B}) det(\pmb{A}) = det(\pmb{B}\pmb{A})$

### Invertible matrix

A square matrix $\pmb{A} \in \mathbb{R}^{N \times N}$ is invertible, and if $\pmb{A}\pmb{A}^{-1} = \pmb{A}^{-1}\pmb{A} = \pmb{I}$.

* $\pmb{A}$ is invertible.
* $\pmb{A}$ is full-rank, i.e.
* The number 0 is not an eigenvalue of $\pmb{A}$.

Time complexity: $O(N^3)$

Notes:
* If you want to know if a matrix $\pmb{A}$ (which is at least PSD) is PD, you can check if it is invertible. If that's the case, then 0 can not be an eigenvalue of $\pmb{A}$ and thus the matrix is PD.

### Orthogonal Matrices

$\pmb{A}$ is an orthogonal (square) matrix if $\pmb{A}^{-1} = \pmb{A}^T$, and thus $\pmb{A}^T\pmb{A} = \pmb{A}\pmb{A}^T = \pmb{I}$. This implies that the columns (resp. rows) of $\pmb{A}$ are othonormal to each other.

### Symmetric Matrices

### Positive (Semi) Definite

* PSD $\rightarrow$ symmetric.

* All the evals of a PD matrix are positive, and 0 is not one of these.
* All the evals of a PSD matrix are non-negative.

### Norm

### Trace

\begin{equation}
    tr(\pmb{A}) = \sum_i a_{ii}
\end{equation}

Properties:
* The trace is a linear operator: $tr(c_1 \pmb{A}+ c_2 \pmb{B}) = c_1 tr(\pmb{A}) + c_2 tr(\pmb{B})$
* The transpose has the same trace: $tr(\pmb{A}) = tr(\pmb{A}^T)$
* Invariance under cyclic permutation: $tr(\pmb{A}\pmb{B}\pmb{C}) = tr(\pmb{B}\pmb{C}\pmb{A}) = tr(\pmb{C}\pmb{A}\pmb{B})$
* Product: $tr(\pmb{A}\pmb{B}) \neq tr(\pmb{A})tr(\pmb{B})$
* Frobenius Norm: $||\pmb{A}||_F = \sqrt{tr(\pmb{A}^T\pmb{A})} = \sqrt{tr(\pmb{A}\pmb{A}^T)}$

### Matrix Calculus

More information can be found on the [matrix cookbook](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf).

First order:
\begin{align}
    \nabla_{\pmb{X}} tr(\pmb{AX}) = \nabla_{\pmb{X}} tr(\pmb{XA}) &= \pmb{A}^T \\
    \nabla_{\pmb{X}} tr(\pmb{AX}^T) = \nabla_{\pmb{X}} tr(\pmb{X}^T\pmb{A}) &= \pmb{A}
\end{align}

Second order:
\begin{align}
    \nabla_{\pmb{X}} tr(\pmb{A}\pmb{X}\pmb{B}\pmb{X}) = \nabla_{\pmb{X}} tr(\pmb{X}\pmb{B}\pmb{X}\pmb{A}) &= (\pmb{BXA})^T + (\pmb{AXB})^T \\
    \nabla_{\pmb{X}} tr(\pmb{A}\pmb{X}^T\pmb{B}\pmb{X}) = \nabla_{\pmb{X}} tr(\pmb{X}^T\pmb{B}\pmb{X}\pmb{A}) = \nabla_{\pmb{X}} tr(\pmb{B}\pmb{X}\pmb{A}\pmb{X}^T) &= \pmb{BXA} + (\pmb{A}\pmb{X}^T\pmb{B})^T \\
\end{align}

### Pseudo inverse

This is the generalization of the inverse, in the sense that it can be applied to rectangular matrices. Assume $\pmb{A} \in \mathbb{R}^{N \times N}$.

Right pseudo-inverse:
\begin{equation}
    \pmb{A}^\dagger = (\pmb{A}^T\pmb{A})^{-1}\pmb{A}^T
\end{equation}

Left pseudo-inverse:
\begin{equation}
    ^\dagger\pmb{A} = \pmb{A}^T(\pmb{A}\pmb{A}^T)^{-1}
\end{equation}

#### Linear Regression (LR)

Assume the input is given by $\pmb{X} \in \mathbb{R}^{N \times D_x}$, the output by $\pmb{Y} \in \mathbb{R}^{N \times D_y}$, and the weight matrix by $\pmb{W} \in \mathbb{R}^{D_x \times D_y}$.

The MSE loss is defined as:

\begin{equation}
    \mathcal{L} = ||\pmb{Y} - \pmb{XW}||^2_F
\end{equation}

Taking the gradient of this loss and setting it to zero gives us the minimum:

\begin{align}
    \nabla_{\pmb{W}} \mathcal{L} &= \nabla_{\pmb{W}} ||\pmb{Y} - \pmb{XW}||^2_F \\
    &= \nabla_{\pmb{W}} tr((\pmb{Y} - \pmb{XW})^T(\pmb{Y} - \pmb{XW})) \\
    &= \nabla_{\pmb{W}} [tr(\pmb{Y}^T\pmb{Y}) - tr((\pmb{XW})^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW}) + tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W})] \\
    &= \nabla_{\pmb{W}} [tr(\pmb{Y}^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW}) - tr(\pmb{Y}^T\pmb{XW}) + tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W})] \\
    &= -2 \nabla_{\pmb{W}} tr(\pmb{Y}^T\pmb{XW}) + \nabla_{\pmb{W}} tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W}) \\
    &= -2 (\pmb{Y}^T\pmb{X})^T + \pmb{X}^T\pmb{X} \pmb{W} + (\pmb{X}^T\pmb{X})^T \pmb{W} \\
    &= -2 \pmb{X}^T\pmb{Y} + 2 \pmb{X}^T\pmb{X} \pmb{W} \\
    &= 0 \\
    \Leftrightarrow & \quad (\pmb{X}^T\pmb{X}) \pmb{W} = \pmb{X}^T\pmb{Y}
\end{align}

If the covariance $(\pmb{X}^T\pmb{X})$ is invertible i.e. is PD, then the best set of weights are given by:

\begin{equation}
    \pmb{W}^* = (\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T\pmb{Y} = \pmb{X}^\dagger \pmb{Y} = \pmb{\Sigma_{XX}}^{-1}\pmb{\Sigma_{XY}}
\end{equation}

#### Linear Weighted Regression (LWR)

### Covariance

The covariance $\pmb{C}_{XY}$ is a positive semi-definite (PSD) matrix which captures linear correlation between 2 random variables $X$ and $Y$. The PSD implies that it is symmetric by definition.

* If the cov is invertible, then 0 can not be an eigenvalue of $\pmb{C}$. This means that it is positive definite (PD).

### Eigenvalue and Eigenvectors

### Diagonalizable and Eigendecomposition

### Idempotence

An ***idempotent*** matrix $\pmb{P}$ is a square 'matrix which, when multiplied by itself, yields itself'.

\begin{equation}
    \pmb{P} = \pmb{PP} = \pmb{P}^2
\end{equation}

Properties:
* An idempotent matrix (except the identity) is singular (i.e. not full rank).
* $\pmb{I} - \pmb{P}$ is also idempotent.
* An idempotent matrix is always diagonalizable and its eigenvalues are either 0 or 1.
* The trace of an idempotent matrix equals the rank of the matrix and thus is always an integer.

In linear regression, the optimal solution of $\mathcal{L} = ||\pmb{Y} - \pmb{XW}||^2_F$ with respect to $\pmb{W}$ is $\pmb{W}^* = (\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T\pmb{Y} = \pmb{X}^\dagger \pmb{Y}$.

The residual error is then given by:

\begin{equation}
    E = (\pmb{Y} - \pmb{XW}^*) = (\pmb{Y} - \pmb{X}(\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T\pmb{Y}) = [\pmb{I} - \pmb{X}(\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T] \pmb{Y} = [\pmb{I} - \pmb{X}\pmb{X}^\dagger] \pmb{Y} = \pmb{Q}\pmb{Y}
\end{equation}

The matrices $\pmb{P} = \pmb{X}\pmb{X}^\dagger$ and $\pmb{Q} = [\pmb{I} - \pmb{X}\pmb{X}^\dagger] = [\pmb{I} - \pmb{P}]$ are symmetric and idempotent matrices.

* An idempotent linear operator $\pmb{P}$ is a projection operator on the range space $\mathcal{R}(\pmb{P})$ along its null space $\mathcal{N}(\pmb{P})$.
* $\pmb{P}$ is an orthogonal projection operator $\Leftrightarrow$ it is idempotent and symmetric.
Ex: $\pmb{P} = \left[\begin{array}{cc} \pmb{I} & \pmb{0} \\ \pmb{0} & \pmb{0} \end{array} \right]$

### Orthogonal Complement and Projection Matrix

"A ***projection*** is a linear transformation $\pmb{P}$ from a vector space $V$ to itself such that $\pmb{P}^2 = \pmb{P}$ (i.e. $\pmb{P}$ is idempotent). That is, whenever $\pmb{P}$ is applied twice to any value, it gives the same result as if it were applied once (idempotent). It leaves its image unchanged.

Let $V$ be a finite dimensional vector space and $\pmb{P}$ be a projection on $V$. Suppose the subspaces $\mathcal{R}$ and $\mathcal{N}$ are the range and null space (aka kernel) of $\pmb{P}$ respectively. Then $\pmb{P}$ has the following properties:
1. $\pmb{P}$ is idempotent by def (i.e. $\pmb{P}^2 = \pmb{P}$)
2. $\pmb{P}$ is the identity operator $\pmb{I}$ on $\mathcal{R}$ (i.e. $\forall \pmb{x} \in \mathcal{R}: \pmb{Px} = \pmb{x}$)
3. We have a direct sum $V = \mathcal{R} \oplus \mathcal{N}$. Every vector $\pmb{x} \in V$ may be decomposed uniquely as $\pmb{x} = \pmb{r} + \pmb{n}$ with $\pmb{r} = \pmb{Px} \in \mathcal{R}$ and $\pmb{n} = \pmb{x} − \pmb{Px} = (\pmb{I} - \pmb{P}) \pmb{x} \in \mathcal{N}$.

The range and kernel of a projection are complementary, as are $\pmb{P}$ and $\pmb{Q} = \pmb{I} − \pmb{P}$. The operator $\pmb{Q}$ is also a projection, and the range and null spaces of $\pmb{P}$ become the null and range spaces of $\pmb{Q}$ and vice versa. We say $\pmb{P}$ is a projection along $\mathcal{N}$ onto $\mathcal{R}$, and $\pmb{Q}$ is a projection along $\mathcal{R}$ onto $\mathcal{N}$."

##### Orthogonal Projection

When the vector space $V$ has an inner product and is complete (i.e. it is a Hilbert space), the concept of orthogonality can be used. An ***orthogonal projection*** is a projection for which the range $\mathcal{R}$ and the null space $\mathcal{N}$ are orthogonal subspaces. That is, $\forall \pmb{r} \in \mathcal{R}, \forall \pmb{n} \in \mathcal{N}: \pmb{r} . \pmb{n} = 0$.

The ***orthogonal complement*** of a subspace $W$ of a vector space $V$ equipped with a bilinear form $B$ is the set $W^\perp$ of all vectors in $V$ that are orthogonal to every vector in $W$.

### SVD

Any rectangular matrices $\pmb{A} \in \mathbb{R}^{N \times M}$ can be decomposed into a product of 3 matrices (which can be seen as the building blocks of $\pmb{A}$):

\begin{equation}
    \pmb{A} = \pmb{U_A}\pmb{\Sigma_A}\pmb{V_A}^T
\end{equation}
where:
* $\pmb{U_A} \in \mathbb{R}^{N \times N}$ is an orthogonal matrix (i.e. $\pmb{U_A}^T = \pmb{U_A}^{-1}$ thus $\pmb{U_A}\pmb{U_A}^T = \pmb{U_A}^T\pmb{U_A} = \pmb{I}$). The columns of $\pmb{U_A}$ contains the eigenvectors of the PSD (thus symmetric) $\pmb{A}\pmb{A}^T$, and are also known as the left-singular vectors of $\pmb{A}$. Its first columns associated with non-zero singular values span the range of $\pmb{A}$. Note that because $\pmb{U_A}$ is an orthogonal matrix, it is invertible and thus has full rank, i.e. its columns form a basis and span $\mathbb{R}^N$.
* $\pmb{\Sigma_A} \in \mathbb{R}^{N \times M}$ is a rectangular matrix. The upper left submatrix is a diagonal matrix of size $r \times r$ with $r = \min(N,M)$ while the rest is filled with zeros. The diagonal elements are the singular values ordered by descending order. The singular values (SVs) $\sigma_i$ are equals to the square roots of the eigenvalues $\lambda_i$ of $\pmb{A}\pmb{A}^T$ and $\pmb{A}^T\pmb{A}$. The rank of $\pmb{A}$ is given by the number of SVs different from 0.
* $\pmb{V_A} \in \mathbb{R}^{M \times M}$ is an orthogonal matrix (i.e. $\pmb{V_A}^T = \pmb{V_A}^{-1}$ thus $\pmb{V_A}\pmb{V_A}^T = \pmb{V_A}^T\pmb{V_A} = \pmb{I}$). The columns of $\pmb{V_A}$ contains the eigenvectors of the PSD (thus symmetric) $\pmb{A}^T\pmb{A}$, and are also known as the right-singular vectors of $\pmb{A}$. Its last columns associated with vanishing singular values $\sigma_i = 0$ span the null space of $\pmb{A}$. Note that because $\pmb{V_A}$ is an orthogonal matrix, it is invertible and thus has full rank, i.e. its columns form a basis and span $\mathbb{R}^M$.

Few notes:
* SVD can be seen as a generalization of eigendecomposition.
* SVD can be compressed such that $\pmb{A} = \pmb{\tilde{U}_A}\pmb{\tilde{\Sigma}_A}\pmb{\tilde{V_A}^T}$ with $\pmb{\tilde{U}_A} \in \mathbb{R}^{N \times r}$, $\pmb{\tilde{\Sigma}_A} \in \mathbb{R}^{r \times r}$, $\pmb{V_A} \in \mathbb{R}^{M \times r}$, and $r=\min(N,M)$.
*

Some properties:
* **Existence**:
* **Uniqueness**:
* **Transpose**: $\pmb{A}^T = (\pmb{U_A}\pmb{\Sigma_A}\pmb{V_A}^T)^T = \pmb{V_A}\pmb{\Sigma_A}^T\pmb{U_A}^T $
* If the matrix A is symmetric, then it has real eigenvalues.
* **Pseudo-inverse**: The pseudo-inverse of $\pmb{A}$ is given by $\pmb{A^\dagger} = \pmb{V_A}\pmb{\Sigma_A}^{-1}\pmb{U_A}^T$ with $\pmb{\Sigma_A}^{-1}$ containing the inverse of singular values on its diagonal.
* **Null and range space** of $\pmb{A}$: $\mathcal{N}(\pmb{A}) \equiv \mathcal{R}^\perp(\pmb{A}^T)$ and $\mathcal{R}(\pmb{A}) \equiv \mathcal{N}^\perp(\pmb{A}^T)$. This can be seen by applying SVD on $\pmb{A}$ and $\pmb{A}^T$. The last columns of $\pmb{V_A}$ associated with vanishing SVs spans the null-space of $\pmb{A}$, and the first columns of $\pmb{V_A}$ associated with non-zero SVs spans the range space of $\pmb{A}^T$. Because of the orthogonality property of $\pmb{V_A}$, we have the $r$ first columns are orthogonals to the $M-r$ last columns. $dim(\mathcal{R}(\pmb{A})) + dim(\mathcal{N}(\pmb{A})) = M$

### Block Matrices

### PCA

The (MSE) loss minimized by PCA is:

\begin{equation}
    \mathcal{L} = ||\pmb{X} - \pmb{XWW}^T||^2_F
\end{equation}

where is the $\pmb{\tilde{X}} = \pmb{XW}$ is the projected data on the lower dimensional space, and $\pmb{\hat{X}} = \pmb{\tilde{X}W}^T$ is the data projected back to the original space.

Taking the gradient of this loss with respect to $\pmb{W}$ and setting it to $0$ gives us the minimum:

\begin{align}
    \nabla_{\pmb{W}} \mathcal{L} &= \nabla_{\pmb{W}} tr((\pmb{X} - \pmb{XWW}^T)^T(\pmb{X} - \pmb{XWW}^T)) \\
    &= \nabla_{\pmb{W}} tr((\pmb{X}^T - \pmb{WW}^T \pmb{X}^T) (\pmb{X} - \pmb{XWW}^T)) \\
    &= \nabla_{\pmb{W}} [tr(\pmb{X}^T\pmb{X}) - tr(\pmb{WW}^T \pmb{X}^T\pmb{X}) - tr(\pmb{X}^T\pmb{X}\pmb{WW}^T) + tr(\pmb{WW}^T\pmb{X}^T\pmb{XWW}^T)] \\
    &= \nabla_{\pmb{W}} [- 2 tr(\pmb{W}^T \pmb{\Sigma_{XX}} \pmb{W}) + tr(\pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{WW}^T)] \\
    &= - 2 (\pmb{\Sigma_{XX}} \pmb{W} + \pmb{\Sigma_{XX}}^T \pmb{W}) + (\pmb{W}^T \pmb{\Sigma_{XX}} \pmb{WW}^T)^T + (\pmb{\Sigma_{XX}} \pmb{WW}^T)\pmb{W} + (\pmb{WW}^T \pmb{\Sigma_{XX}})^T (\pmb{W}^T)^T + (\pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W}) \\
    &= - 4 \pmb{\Sigma_{XX}} \pmb{W} + \pmb{WW}^T \pmb{\Sigma_{XX}}^T \pmb{W} + \pmb{\Sigma_{XX}} \pmb{WW}^T\pmb{W} + \pmb{\Sigma_{XX}}^T\pmb{WW}^T \pmb{W} + \pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W} \\
    &= - 4 \pmb{\Sigma_{XX}} \pmb{W} + 2 \pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W} + 2 \pmb{\Sigma_{XX}} \pmb{WW}^T\pmb{W} \\
    &= 0 \\
    \Leftrightarrow & \quad \pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W} + \pmb{\Sigma_{XX}} \pmb{WW}^T\pmb{W} = 2  \pmb{\Sigma_{XX}} \pmb{W} \\
    \Leftrightarrow & \quad \pmb{WW}^T \pmb{\Sigma_{XX}} + \pmb{\Sigma_{XX}} \pmb{WW}^T = 2 \pmb{\Sigma_{XX}} \\
    \Leftrightarrow & \quad \pmb{WW}^T + \pmb{\Sigma_{XX}} \pmb{WW}^T \pmb{\Sigma_{XX}}^{-1} = 2 \pmb{I} \\
    \Leftrightarrow & \quad \pmb{\Sigma_{XX}} = (2 \pmb{I} - \pmb{WW}^T) \pmb{\Sigma_{XX}} (\pmb{WW}^T)^{-1} \\
    \Leftrightarrow & \quad \pmb{Q\Lambda Q}^T = (2 \pmb{I} - \pmb{WW}^T) \pmb{Q\Lambda Q}^T (\pmb{WW}^T)^{-1} \\
    \Leftrightarrow & \quad \left\{ \begin{array}{l} (2 \pmb{I} - \pmb{WW}^T) \pmb{Q} = \pmb{Q} \\ \pmb{Q}^T (\pmb{WW}^T)^{-1} = \pmb{Q}^T \: \Leftrightarrow \: \pmb{QQ}^T = \pmb{WW}^T \end{array} \right.  \\
\end{align}

Algo using the covariance:
1. subtract the mean $\pmb{X}$
2. compute the covariance matrix $\pmb{\Sigma_{XX}} = \pmb{X}^T\pmb{X}$
3. compute the eigendecomposition of $\pmb{\Sigma_{XX}}$, i.e. compute $\pmb{Q}$ and $\pmb{\Lambda}$ such that $\pmb{\Sigma_{XX}} = \pmb{Q} \pmb{\Lambda} \pmb{Q}^T$.
4. return the sorted evals and the corresponding evecs

Algo using SVD:
1. substract the mean from $\pmb{X}$
2. compute SVD of $\pmb{X}$, i.e. $\pmb{X} = \pmb{U_X \Sigma_X V_X}^T$. The eigenvectors are given by $\pmb{Q} = \pmb{V_X}$ and the evals by $\pmb{\Lambda} = \pmb{\Sigma_X}^2$.

#### Recursive PCA/SVD

#### Hierarchical PCA/SVD

### Linear FNN with 1 hidden layer

Assume a linear Feedforward Neural Network (FNN) with 1 input, 1 hidden, and 1 output layer. Assume the input data is given by $\pmb{X} \in \mathbb{R}^{N \times D_x}$, the output data by $\pmb{Y} \in \mathbb{R}^{N \times D_y}$, the hidden data by $\pmb{H} \in \mathbb{R}^{N \times D_h}$, the input-hidden weight matrix by $\pmb{W_1} \in \mathbb{R}^{D_x \times D_h}$ and the hidden-output weight matrix $\pmb{W_2} \in \mathbb{R}^{D_h \times D_y}$. These variables are related by the following relationships:

\begin{equation}
    \pmb{H} = \pmb{X} \pmb{W_1} \qquad \mbox{and} \qquad \pmb{Y} = \pmb{H} \pmb{W_2}
\end{equation}

The MSE loss is thus defined as:

\begin{equation}
    \mathcal{L} = ||\pmb{Y} - \pmb{XW_1W_2}||^2_F
\end{equation}

Taking the gradient of this loss with respect to $\pmb{W_1}$ and $\pmb{W_2}$, and setting these to zero gives us the minimum:

\begin{align}
    \nabla_{\pmb{W_1}} \mathcal{L} &= \nabla_{\pmb{W_1}} ||\pmb{Y} - \pmb{XW_1W_2}||^2_F \\
    &= \nabla_{\pmb{W_1}} tr((\pmb{Y} - \pmb{XW_1W_2})^T(\pmb{Y} - \pmb{XW_1W_2})) \\
    &= \nabla_{\pmb{W_1}} [tr(\pmb{Y}^T\pmb{Y}) - tr((\pmb{XW_1W_2})^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW_1W_2}) + tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W})] \\
    &= \nabla_{\pmb{W_1}} [tr(\pmb{Y}^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW_1W_2}) - tr(\pmb{Y}^T\pmb{XW_1W_2}) + tr(\pmb{(W_1W_2)}^T\pmb{X}^T\pmb{X}\pmb{(W_1W_2)})] \\
    &= \nabla_{\pmb{W_1}} [-2 tr(\pmb{Y}^T\pmb{XW_1W_2}) + tr(\pmb{W_2}^T\pmb{W_1}^T\pmb{X}^T\pmb{X}\pmb{W_1}\pmb{W_2})] \\
    &= \nabla_{\pmb{W_1}} [-2 tr(\pmb{\Sigma_{YX}}\pmb{W_1W_2}) + tr(\pmb{W_2}^T\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2})] \\
    &= -2 (\pmb{W_2}\pmb{\Sigma_{YX}})^T + \pmb{\Sigma_{XX}}\pmb{W_1W_2W_2}^T + \pmb{\Sigma_{XX}}^T\pmb{W_1}(\pmb{W_2W_2}^T)^T \\
    &= -2 \pmb{\Sigma_{XY}} \pmb{W_2}^T + 2 \pmb{\Sigma_{XX}}\pmb{W_1W_2W_2}^T \\
    &= 0 \\
    \Leftrightarrow & \quad \pmb{\Sigma_{XX}}\pmb{W_1W_2W_2}^T = \pmb{\Sigma_{XY}} \pmb{W_2}^T \\
    \Leftrightarrow & \quad \pmb{W_1} = \pmb{\Sigma_{XX}}^{-1} \pmb{\Sigma_{XY}} \pmb{W_2}^T (\pmb{W_2W_2}^T)^{-1}
\end{align}

and

\begin{align}
    \nabla_{\pmb{W_2}} \mathcal{L} &= \nabla_{\pmb{W_2}} [-2 tr(\pmb{\Sigma_{YX}}\pmb{W_1W_2}) + tr(\pmb{W_2}^T\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2})] \\
    &= -2 (\pmb{\Sigma_{YX}}\pmb{W_1})^T + \pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2} + (\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1})^T\pmb{W_2} \\
    &= -2 \pmb{W_1}^T\pmb{\Sigma_{XY}} + 2 \pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2} \\
    &= 0 \\
    \Leftrightarrow & \quad \pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2} = \pmb{W_1}^T \pmb{\Sigma_{XY}} \\
    \Leftrightarrow & \quad \pmb{W_2} = (\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1})^{-1} \pmb{W_1}^T \pmb{\Sigma_{XY}} \\
\end{align}

```python

```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function
from __future__ import absolute_import

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Differentiation

**GOAL:**  Given a set of $N+1$ points $(x_i, y_i)$ compute the derivative of a given order to a specified accuracy.

**Approaches:**
 * Find the interpolating polynomial $P_N(x)$ and differentiate that.
 * Use Taylor-series expansions and the method of undetermined coefficients to derive finite-difference weights and their error estimates

**Issues:**  Order vs accuracy...how to choose

# Example 1:  how to approximate the derivative $f'(x)$ given a discrete sampling of a function $f(x)$

Here we will consider how to estimate $f'(x_k)$ given a $N$ point sampling of $f(x)=\sin(\pi x) + 1/2 \sin(2\pi x)$ sampled uniformly over the interval $x\in [ 0,1]$

```python
N = 11
x = numpy.linspace(0,1,N)
xfine = numpy.linspace(0,1,101)
f = lambda x:  numpy.sin(numpy.pi*x) + 0.5*numpy.sin(4*numpy.pi*x)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(xfine, f(xfine),'b',label='$f(x)$')
axes.plot(x, f(x), 'ro', markersize=12, label='$f(x_k)$')
axes.grid()
axes.set_xlabel('x')
p = numpy.polyfit(x,f(x),N-1)
axes.plot(xfine,numpy.polyval(p,xfine),'g--',label='$P_{{{N}}}$'.format(N=N-1))
axes.legend(fontsize=15)
plt.show()
```

### Example 2:  how to approximate derivative $f'(x)$ given a discrete sampling of a function $f(x)$

Here we will consider how to estimate $f'(x_k)$ given a $N$ point sampling of Runge's function sampled uniformly over the interval $x\in [ -1,1]$

```python
N = 11
x = numpy.linspace(-1,1,N)
xfine = numpy.linspace(-1,1,101)
f = lambda x:  1./(1. + 25*x**2)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(xfine, f(xfine),'b',label='$f(x)$')
axes.plot(x, f(x), 'ro', markersize=12, label='$f(x_k)$')
axes.grid()
axes.set_xlabel('x')
p = numpy.polyfit(x,f(x),N-1)
axes.plot(xfine,numpy.polyval(p,xfine),'g--',label='$P_{{{N}}}$'.format(N=N-1))
axes.legend(fontsize=15)
plt.show()
```

### The interpolating polynomial: review

From our previous lecture, we showed that we can approximate a function $f(x)$ over some interval in terms of a unique interpolating polynomial through $N+1$ points and a  remainder term

$$
    f(x) = P_N(x) + R_N(x)
$$

Where the Lagrange remainder term is

$$R_N(x) = (x - x_0)(x - x_1)\cdots (x - x_{N})(x - x_{N+1}) \frac{f^{(N+1)}(c)}{(N+1)!}$$

While there are multiple ways to represent the interpolating polynomial, both $P_N(x)$ and $R_N(x)$ are polynomials in $x$ and therefore differentiable.  Thus we should be able to calculate the first derivative and its error as

$$
    f'(x) = P'_N(x) + R'_N(x)
$$

and likewise for higher order derivatives up to degree $N$.

### Derivatives of the Lagrange Polynomials

The Lagrange basis, is a particularly nice basis for calculating numerical differentiation formulas because of their basic interpolating property that

$$
    P_N(x) = \sum_{i=0}^N f(x_i)\ell_i(x)
$$

where $f(x_i)$ is just the value of our function $f$ at node $x_i$ and all of the $x$ dependence is contained in the Lagrange Polynomials $\ell_i(x)$ (which only depend on the node coordinates $x_i$, $i=0,\ldots,N$).  Thus, the interpolating polynomial at any $x$ is simply a linear combination of the values at the nodes $f(x_i)$

Likewise its first derivative
$$
P'_N(x)  = \sum_{i=0}^N f(x_i)\ell'_i(x)
$$
is also just a linear combination of the values $f(x_i)$

## Examples

Given the potentially, highly oscillatory nature of the interpolating polynomial, in practice we only use a small number of data points around a given point $x_k$ to derive a differentiation formula for the derivative $f'(x_k)$.  In the context of differential equations we also often have $f(x)$ so that $f(x_k) = y_k$ and we can approximate the derivative of a known function $f(x)$.

```python
N = 9
f = lambda x:  1./(1. + 25*x**2)
#f = lambda x: numpy.cos(2.*numpy.pi*x)
```

```python
x = numpy.linspace(-1,1,N)
xfine = numpy.linspace(-1,1,101)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(xfine, f(xfine),'b',label='$f(x)$')
axes.plot(x, f(x), 'ro', markersize=12, label='$f(x_k)$')
x3 = x[5:8]
x3fine = numpy.linspace(x3[0],x3[-1],20)
p = numpy.polyfit(x3,f(x3),2)
axes.plot(x3,f(x3),'m',label = 'Piecewise $P_1(x)$')
axes.plot(x3fine,numpy.polyval(p,x3fine),'k',label = 'Piecewise $P_2(x)$')
axes.grid()
axes.set_xlabel('x')
p = numpy.polyfit(x,f(x),N-1)
axes.plot(xfine,numpy.polyval(p,xfine),'g--',label='$P_{{{N}}}$'.format(N=N-1))
axes.legend(fontsize=14,loc='best')
plt.show()
```

### Example: 1st order polynomial through 2 points $x=x_0, x_1$:

$$
    P_1(x)=f_0\ell_0(x) + f_1\ell_1(x)
$$

Or written out in full

$$
P_1(x) = f_0\frac{x-x_1}{x_0-x_1} + f_1\frac{x-x_0}{x_1-x_0}
$$

Thus the first derivative of this polynomial for all $x\in[x_0,x_1]$ is

$$
P'_1(x) = \frac{f_0}{x_0-x_1} + \frac{f_1}{x_1-x_0} = \frac{f_1 - f_0}{x_1 - x_0} = \frac{f_1 - f_0}{\Delta x}
$$

Where $\Delta x$ is the width of the interval.  This formula is simply the slope of the chord connecting the points $(x_0, f_0)$ and $(x_1,f_1)$.   Note also, that the estimate of the first-derivative is constant for all $x\in[x_0,x_1]$.

#### "Forward" and "Backward" first derivatives

Even though the first derivative by this method is the same at both $x_0$ and $x_1$, we sometime make a distinction between the "forward Derivative"

$$f'(x_n) \approx D_1^+ = \frac{f(x_{n+1}) - f(x_n)}{\Delta x}$$

and the "backward" finite-difference as

$$f'(x_n) \approx D_1^- = \frac{f(x_n) - f(x_{n-1})}{\Delta x}$$

Note these approximations should be familiar to use as the limit as $\Delta x \rightarrow 0$ these are no longer approximations but equivalent definitions of the derivative at $x_n$.

### Example: 2nd order polynomial through 3 points $x=x_0, x_1, x_2$:

$$
    P_2(x)=f_0\ell_0(x) + f_1\ell_1(x) + f_2\ell_2(x)
$$

Or written out in full

$$
P_2(x) = f_0\frac{(x-x_1)(x-x_2)}{(x_0-x_1)(x_0-x_2)} + f_1\frac{(x-x_0)(x-x_2)}{(x_1-x_0)(x_1-x_2)} + f_2\frac{(x-x_0)(x-x_1)}{(x_2-x_0)(x_2-x_1)}
$$

Thus the first derivative of this polynomial for all $x\in[x_0,x_2]$ is

$$
P'_2(x) = f_0\frac{(x-x_1)+(x-x_2)}{(x_0-x_1)(x_0-x_2)} + f_1\frac{(x-x_0)+(x-x_2)}{(x_1-x_0)(x_1-x_2)} + f_2\frac{(x-x_0)+(x-x_1)}{(x_2-x_0)(x_2-x_1)}
$$

**Exercise**: show that the second-derivative $P''_2(x)$ is a constant (find it!) but is also just a linear combination of the function values at the nodes.

### Special case of equally spaced nodes $x = [-h, 0, h]$ where $h=\Delta x$ is the grid spacing

General Case:
$$
P'_2(x) = f_0\frac{(x-x_1)+(x-x_2)}{(x_0-x_1)(x_0-x_2)} + f_1\frac{(x-x_0)+(x-x_2)}{(x_1-x_0)(x_1-x_2)} + f_2\frac{(x-x_0)+(x-x_1)}{(x_2-x_0)(x_2-x_1)}
$$

Becomes:
$$
P'_2(x) = f_0\frac{2x-h}{2h^2} + f_1\frac{-2x}{h^2} + f_2\frac{2x+h}{2h^2}
$$

which if we evaluate at the three nodes  $-h,0,h$ yields

$$
P'_2(-h) = \frac{-3f_0 + 4f_1 -1f_2}{2h}, \quad\quad P'_2(0) = \frac{-f_0 + f_2}{2h}, \quad\quad P'_2(h) = \frac{f_0 -4f_1 + 3f_2}{2h}
$$

Again, just  linear combinations of the values at the nodes $f(x_i)$

#### Quick Checks

In general,  all finite difference formulas can be written as linear combinations of the values of $f(x)$ at the nodes.  The formula's can be hard to remember, but they are easy to check.

* The sum of the coefficients must add to zero.  Why?
* The sign of the coefficients can be checked by inserting $f(x_i) = x_i$

##### Example

Given
$$
P'_2(-h) =\frac{-3f_0 + 4f_1 -1f_2}{2h}
$$

What is $P'_2(-h)$ if

* $$f_0=f_1=f_2$$
* $$f_0 = 0, ~f_1 = 1, ~f_2 = 2$$

### Error Analysis

In addition to calculating finite difference formulas, we can also estimate the error

From Lagrange's Theorem,  the remainder term looks like

$$R_N(x) = (x - x_0)(x - x_1)\cdots (x - x_{N})(x - x_{N+1}) \frac{f^{(N+1)}(c)}{(N+1)!}$$

Thus the derivative of the remainder term $R_N(x)$ is

$$R_N'(x) = \left(\sum^{N}_{i=0} \left( \prod^{N}_{j=0,~j\neq i} (x - x_j) \right )\right ) \frac{f^{(N+1)}(c)}{(N+1)!}$$

The remainder term contains a sum of $N$'th order polynomials and can be awkward to evaluate, however, if we restrict ourselves to the error at any given node $x_k$, the remainder simplifies to

$$R_N'(x_k) = \left( \prod^{N}_{j=0,~j\neq k} (x_k - x_j) \right) \frac{f^{(N+1)}(c)}{(N+1)!}$$

If we let $\Delta x = \max_i |x_k - x_i|$ we then know that the remainder term will be $\mathcal{O}(\Delta x^N)$ as $\Delta x \rightarrow 0$ thus showing that this approach converges and we can find arbitrarily high order approximations (ignoring floating point error).

### Examples

#### First order differences $N=1$

For our first order finite differences, the error term is simply

$$R_1'(x_0) = -\Delta x \frac{f''(c)}{2}$$
$$R_1'(x_1) = \Delta x \frac{f''(c)}{2}$$

Both of which are $O(\Delta x f'')$

#### Second order differences $N=2$

For general second order polynomial interpolation, the derivative of the remainder term is

$$\begin{aligned}
    R_2'(x) &= \left(\sum^{2}_{i=0} \left( \prod^{2}_{j=0,~j\neq i} (x - x_j) \right )\right ) \frac{f'''(c)}{3!} \\
    &= \left ( (x - x_{i+1}) (x - x_{i-1}) + (x-x_i) (x-x_{i-1}) + (x-x_i)(x-x_{i+1}) \right ) \frac{f'''(c)}{3!}
\end{aligned}$$

Again evaluating this expression at the center point $x = x_i$ and assuming evenly space points we have

$$R_2'(x_i) = -\Delta x^2 \frac{f'''(c)}{3!}$$

showing that our error is $\mathcal{O}(\Delta x^2)$.

### Caution

High order does not necessarily imply high-accuracy!

As always, the question remains as to whether the underlying function is well approximated by a high-order polynomial.

### Convergence

Nevertheless, we can always check to see if the error reduces as expected as $\Delta x\rightarrow 0$.  Here we estimate the 1st and 2nd order first-derivative for evenly spaced points

```python
def D1_p(func, x_min, x_max, N):
    """ calculate consistent 1st order Forward difference of a function func(x) defined on the interval [x_min,xmax]
    and sampled at N evenly spaced points"""

    x = numpy.linspace(x_min, x_max, N)
    f = func(x)
    dx = x[1] - x[0]
    f_prime = numpy.zeros(N)
    f_prime[0:-1] = (f[1:] - f[0:-1])/dx
    # and patch up the end point with a backwards difference
    f_prime[-1] = f_prime[-2]

    return f_prime

def D1_2(func, x_min, x_max, N):
    """ calculate consistent 2nd order first derivative of a function func(x) defined on the interval [x_min,xmax]
    and sampled at N evenly spaced points"""

    x = numpy.linspace(x_min, x_max, N)
    f = func(x)
    dx = x[1] - x[0]
    f_prime = numpy.zeros(N)
    f_prime[0] = f[:3].dot(numpy.array([-3, 4, -1]))/(2*dx)
    f_prime[1:-1] = (f[2:N] - f[0:-2])/(2*dx)
    f_prime[-1] = f[-3:].dot(numpy.array([1, -4, 3]))/(2*dx)

    return f_prime
```

#### Note:

This first derivative operator can also be written as a Matrix $D$ such that $f'(\mathbf{x}) = Df(\mathbf{x})$ where $\mathbf{x}$ is a vector of $x$ coordinates. (exercise left for the homework)

```python
N = 11
xmin = 0.
xmax = 1.
func = lambda x:  numpy.sin(numpy.pi*x) + 0.5*numpy.sin(4*numpy.pi*x)
func_prime = lambda x: numpy.pi*numpy.cos(numpy.pi*x) + 2.*numpy.pi * numpy.cos(4*numpy.pi*x)
D1f = D1_p(func, xmin, xmax, N)
D2f = D1_2(func, xmin, xmax, N)
```

```python
xa = numpy.linspace(xmin, xmax, 100)
xi = numpy.linspace(xmin, xmax, N)
fig = plt.figure(figsize=(16, 6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D1f, 'ko',label='$D^+_1(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()

axes = fig.add_subplot(1, 2, 2)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D2f, 'go',label='$D_1^2(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()
plt.show()
```

```python
N = 11
xmin = -1
xmax = 1.
func = lambda x:  1./(1 + 25.*x**2)
func_prime = lambda x: -50. * x / (1. + 25.*x**2)**2
D1f = D1_p(func, xmin, xmax, N)
D2f = D1_2(func, xmin, xmax, N)
```

```python
xa = numpy.linspace(xmin, xmax, 100)
xi = numpy.linspace(xmin, xmax, N)
fig = plt.figure(figsize=(16, 6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D1f, 'ko',label='$D^+_1(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()

axes = fig.add_subplot(1, 2, 2)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D2f, 'go',label='$D_1^2(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()
plt.show()
```

#### Computing Order of Convergence

Say we had the error $E(\Delta x)$ and we wanted to make a statement about the rate of convergence (note we can replace $E$ here with the $R$ from above).  Then we can do the following:
$$\begin{aligned}
    E(\Delta x) &= C \Delta x^n \\
    \log E(\Delta x) &= \log C + n \log \Delta x
\end{aligned}$$

The slope of the line is $n$ when modeling the error like this!  We can also match the first point by solving for $C$:

$$
    C = e^{\log E(\Delta x) - n \log \Delta x}
$$

```python
# Compute the error as a function of delta_x
N_range = numpy.logspace(1, 4, 10, dtype=int)
delta_x = numpy.empty(N_range.shape)
error = numpy.empty((N_range.shape[0], 4))
for (i, N) in enumerate(N_range):
    x_hat = numpy.linspace(xmin, xmax, N)
    delta_x[i] = x_hat[1] - x_hat[0]

    # Compute forward difference
    D1f = D1_p(func, xmin, xmax, N)

    # Compute 2nd order difference
    D2f = D1_2(func, xmin, xmax, N)

    # Calculate the infinity norm or maximum error
    error[i, 0] = numpy.linalg.norm(numpy.abs(func_prime(x_hat) - D1f), ord=numpy.inf)
    error[i, 1] = numpy.linalg.norm(numpy.abs(func_prime(x_hat) - D2f), ord=numpy.inf)

error = numpy.array(error)
delta_x = numpy.array(delta_x)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.loglog(delta_x, error[:,0], 'ro', label='$D_1^+$')
axes.loglog(delta_x, error[:,1], 'bo', label='$D_1^2$')
axes.loglog(delta_x, order_C(delta_x[0], error[0, 0], 1.0) * delta_x**1.0, 'r--', label="1st Order")
axes.loglog(delta_x, order_C(delta_x[0], error[0, 1], 2.0) * delta_x**2.0, 'b--', label="2nd Order")
axes.legend(loc=4)
axes.set_title("Convergence of Finite Differences", fontsize=18)
axes.set_xlabel("$\Delta x$", fontsize=16)
axes.set_ylabel("$|f'(x) - \hat{f}'(x)|$", fontsize=16)
axes.legend(loc='best', fontsize=14)
axes.grid()

plt.show()
```

# Another approach:  The method of undetermined Coefficients

An alternative method for finding finite-difference formulas is by using Taylor series expansions about the point we want to approximate.  The Taylor series about $x_n$ is

$$f(x) = f(x_n) + (x - x_n) f'(x_n) + \frac{(x - x_n)^2}{2!} f''(x_n) + \frac{(x - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x - x_n)^4)$$

Say we want to derive the second order accurate, first derivative approximation that we just did, this requires the values $(x_{n+1}, f(x_{n+1})$ and $(x_{n-1}, f(x_{n-1})$.  We can express these values via our Taylor series approximation above as

\begin{aligned}
    f(x_{n+1}) &= f(x_n) + (x_{n+1} - x_n) f'(x_n) + \frac{(x_{n+1} - x_n)^2}{2!} f''(x_n) + \frac{(x_{n+1} - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x_{n+1} - x_n)^4) \\
\end{aligned}

or
\begin{aligned}
&= f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{aligned}

and

\begin{align}
f(x_{n-1}) &= f(x_n) + (x_{n-1} - x_n) f'(x_n) + \frac{(x_{n-1} - x_n)^2}{2!} f''(x_n) + \frac{(x_{n-1} - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x_{n-1} - x_n)^4)
\end{align}

\begin{align}
&= f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{align}

Or all together (for regularly spaced points),
\begin{align}
f(x_{n+1}) &= f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\\
f(x_n) &= f(x_n) \\
f(x_{n-1})&= f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{align}

Now to find out how to combine these into an expression for the derivative we assume our approximation looks like

$$
    f'(x_n) + R(x_n) = A f(x_{n+1}) + B f(x_n) + C f(x_{n-1})
$$

where $R(x_n)$ is our error.

Plugging in the Taylor series approximations we find

$$\begin{aligned}
    f'(x_n) + R(x_n) &= A \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right ) \\
    & + B  ~~~~f(x_n)  \\
    & + C \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4) \right )
\end{aligned}$$

Or
$$
f'(x_n) + R(x_n)= (A + B + C) f(x_n) + (A\Delta x +0B - C\Delta x)f'(x_n) + (A\frac{\Delta x^2}{2!} + C\frac{\Delta x^2}{2!})f''(x_n) + O(\Delta x^3)
$$

Since we want $R(x_n) = \mathcal{O}(\Delta x^2)$ we want all terms lower than this to cancel except for those multiplying $f'(x_n)$ as those should sum to 1 to give us our approximation.  Collecting the terms with common evaluations of the derivatives on $f(x_n)$ we get a series of expressions for the coefficients $A$, $B$, and $C$ based on the fact we want an approximation to $f'(x_n)$.  The $n=0$ terms collected are $A + B + C$ and are set to 0 as we want the $f(x_n)$ term to also cancel.

$$\begin{aligned}
    f(x_n):&  &A + B + C &= 0 \\
    f'(x_n): & &A \Delta x - C \Delta x &= 1 \\
    f''(x_n): & &A \frac{\Delta x^2}{2} + C \frac{\Delta x^2}{2} &= 0
\end{aligned} $$

Or as a linear algebra problem

$$\begin{bmatrix}
1 & 1 & 1 \\
\Delta x & 0 &-\Delta x \\
\frac{\Delta x^2}{2} & 0 & \frac{\Delta x^2}{2} \\
\end{bmatrix}
\begin{bmatrix} A \\ B\\ C\\\end{bmatrix} =
\begin{bmatrix} 0 \\ 1\\ 0\\\end{bmatrix}
$$

This last equation $\Rightarrow A = -C$, using this in the second equation gives $A = \frac{1}{2 \Delta x}$ and $C = -\frac{1}{2 \Delta x}$.  The first equation then leads to $B = 0$.

Putting this altogether then gives us our previous expression including an estimate for the error:

$$\begin{aligned}
    f'(x_n) + R(x_n) &= \quad \frac{1}{2 \Delta x} \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right ) \\
    & \quad + 0 \cdot f(x_n) \\
    & \quad - \frac{1}{2 \Delta x} \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4) \right ) \\
    &=  f'(x_n) + \frac{1}{2 \Delta x} \left ( \frac{2 \Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right )
\end{aligned}$$
so that we find
$$
    R(x_n) = \frac{\Delta x^2}{3!} f'''(x_n) + \mathcal{O}(\Delta x^3) = \mathcal{O}(\Delta x^2)
$$

#### Another way...

There is one more way to derive the second order accurate, first order finite-difference formula.  Consider the two first order forward and backward finite-differences averaged together:

$$\frac{D_1^+(f(x_n)) + D_1^-(f(x_n))}{2} = \frac{f(x_{n+1}) - f(x_n) + f(x_n) - f(x_{n-1})}{2 \Delta x} = \frac{f(x_{n+1}) - f(x_{n-1})}{2 \Delta x}$$

### Example 4: Higher Order Derivatives

Using our Taylor series approach lets derive the second order accurate second derivative formula.  Again we will use the same points and the Taylor series centered at $x = x_n$ so we end up with the same expression as before:

$$\begin{aligned}
    f''(x_n) + R(x_n) &= \quad A \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5)\right ) \\
    &+ \quad B \cdot f(x_n) \\
    &+ \quad C \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right )
\end{aligned}$$

except this time we want to leave $f''(x_n)$ on the right hand side.

Try out the same trick as before and see if you can setup the equations that need to be solved.

Doing the same trick as before we have the following expressions:

$$\begin{aligned}
    f(x_n): & & A + B + C &= 0\\
    f'(x_n): & & A \Delta x - C \Delta x &= 0\\
    f''(x_n): & & A \frac{\Delta x^2}{2} + C \frac{\Delta x^2}{2} &= 1
\end{aligned}$$

Or again

$$\begin{bmatrix}
1 & 1 & 1 \\
\Delta x & 0 &-\Delta x \\
\frac{\Delta x^2}{2} & 0 & \frac{\Delta x^2}{2} \\
\end{bmatrix}
\begin{bmatrix} A \\ B\\ C\\\end{bmatrix} =
\begin{bmatrix} 0 \\ 0\\ 1\\\end{bmatrix}
$$

Note,  the Matrix remains, the same, only the right hand side has changed

The second equation implies $A = C$ which combined with the third implies

$$A = C = \frac{1}{\Delta x^2}$$

Finally the first equation gives

$$B = -\frac{2}{\Delta x^2}$$

leading to the final expression

$$\begin{aligned}
    f''(x_n) + R(x_n) &= \quad \frac{1}{\Delta x^2} \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5)\right ) \\
    &+ \quad -\frac{2}{\Delta x^2} \cdot f(x_n) \\
    &+ \quad \frac{1}{\Delta x^2} \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right ) \\
    &= f''(x_n) + \frac{1}{\Delta x^2} \left(\frac{2 \Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right )
\end{aligned}
$$
so that

$$
    R(x_n) = \frac{\Delta x^2}{12} f^{(4)}(x_n) + \mathcal{O}(\Delta x^3)
$$

```python
def D2(func, x_min, x_max, N):
    """ calculate consistent 2nd order second derivative of a function func(x) defined on the interval [x_min,xmax]
    and sampled at N evenly spaced points"""

    x = numpy.linspace(x_min, x_max, N)
    f = func(x)
    dx = x[1] - x[0]
    D2f = numpy.zeros(x.shape)
    D2f[1:-1] = (f[:-2] - 2*f[1:-1] + f[2:])/(dx**2)
    # patch up end points to be 1 sided 2nd derivatives
    D2f[0] = D2f[1]
    D2f[-1] = D2f[-2]

    return D2f
```

```python
f = lambda x: numpy.sin(x)
f_dubl_prime = lambda x: -numpy.sin(x)

# Use uniform discretization
x = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, 1000)
N = 80
x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
delta_x = x_hat[1] - x_hat[0]

# Compute derivative
D2f  = D2(f, x_hat[0], x_hat[-1], N)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x,f(x),'b',label='$f(x)$')
axes.plot(x, f_dubl_prime(x), 'k--', label="$f'(x)$")
axes.plot(x_hat, D2f, 'ro', label='$D_2(f)$')
axes.set_xlim((x[0], x[-1]))
axes.set_ylim((-1.1, 1.1))
axes.legend(loc='best',fontsize=14)
axes.grid()
axes.set_title('Discrete Second derivative',fontsize=18)
axes.set_xlabel('$x$', fontsize=16)

plt.show()
```

### The general case

In the general case we can use any $N+1$ points to calculated consistent finite difference coefficients for approximating any derivative of order $k \leq N$.  Relaxing the requirement of equal grid spacing (or the expectation that the location where the derivative is evaluated  $\bar{x}$, is one of the grid points) the Taylor series expansions become

$$\begin{aligned}
    f^{(k)}(\bar{x}) + R(\bar{x}) &= \quad c_0 \left ( f(\bar{x}) + \Delta x_0 f'(\bar{x}) + \frac{\Delta x_0^2}{2!} f''(\bar{x}) + \frac{\Delta x_0^3}{3!} f'''(\bar{x}) + \frac{\Delta x_0^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_0^5)\right ) \\
    &+ \quad c_1 \left ( f(\bar{x}) + \Delta x_1 f'(\bar{x}) + \frac{\Delta x_1^2}{2!} f''(\bar{x}) + \frac{\Delta x_1^3}{3!} f'''(\bar{x}) + \frac{\Delta x_1^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_1^5)\right )\\
    &+ \quad c_2 \left ( f(\bar{x}) + \Delta x_2 f'(\bar{x}) + \frac{\Delta x_2^2}{2!} f''(\bar{x}) + \frac{\Delta x_2^3}{3!} f'''(\bar{x}) + \frac{\Delta x_2^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_2^5)\right ) \\
    &+ \quad \vdots\\
    &+ \quad c_N \left ( f(\bar{x}) + \Delta x_N f'(\bar{x}) + \frac{\Delta x_N^2}{2!} f''(\bar{x}) + \frac{\Delta x_N^3}{3!} f'''(\bar{x}) + \frac{\Delta x_N^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_N^5)\right ) \\
\end{aligned}$$
where $\Delta\mathbf{x} = \bar{x} - \mathbf{x}$ is the distance between the point $\bar{x}$ and each grid point.

Equating terms of equal order reduces the problem to another Vandermonde matrix problem
$$\begin{bmatrix}
1 & 1 & 1 & \cdots & 1 \\
\Delta x_0 & \Delta x_1 & \Delta x_2 & \cdots & \Delta x_N\\
\frac{\Delta x_0^2}{2!} & \frac{\Delta x_1^2}{2!} & \frac{\Delta x_2^2}{2!} &\cdots &  \frac{\Delta x_N^2}{2!}\\
 &  & \vdots & \cdots &  \\
\frac{\Delta x_0^N}{N!} & \frac{\Delta x_1^N}{N!} & \frac{\Delta x_2^N}{N!} & \cdots &  \frac{\Delta x_N^N}{N!}\\
\end{bmatrix}
\begin{bmatrix} c_0 \\ c_1\\ c_2 \\ \vdots \\ c_N\\\end{bmatrix} =
\mathbf{b}_k
$$

where $\mathbf{b}_k$ is a vector of zeros with just a one in the $k$th position for the $k$th derivative.

By exactly accounting for the first $N+1$ terms of the Taylor series (with $N+1$ equations),  we can get any order derivative $0<k<N$ as well as an Error estimate for

$$R(\bar{x}) = O\left(\frac{\mathbf{c}^T\Delta\mathbf{x}^{N+1}}{(N+1)!}f^{(N+1)}\right) $$

This approach of "undetermined coefficients" can be efficiently coded up as a routine to provide consistent $Nth$ order finite difference coefficients for an arbitrarily spaced grid $\mathbf{x}$.

Here we present a python version of a matlab routine `fdcoeffV.m` from Randy Leveque's excellent book [Finite Difference Methods for ordinary and partial differential equations](https://faculty.washington.edu/rjl/fdmbook/)

```python
def fdcoeffV(k,xbar,x):
    """
    fdcoeffV routine modified from Leveque (2007) matlab function

    Params:
    -------

    k: int
        order of derivative
    xbar: float
        point at which derivative is to be evaluated
    x: ndarray
        numpy array of coordinates to use in calculating the weights

    Returns:
    --------
    c: ndarray
        array of floats of coefficients.

    Compute coefficients for finite difference approximation for the
    derivative of order k at xbar based on grid values at points in x.

    WARNING: This approach is numerically unstable for large values of n since
    the Vandermonde matrix is poorly conditioned.  Use fdcoeffF.m instead,
    which is based on Fornberg's method.

     This function returns a row vector c of dimension 1 by n, where n=length(x),
     containing coefficients to approximate u^{(k)}(xbar),
     the k'th derivative of u evaluated at xbar,  based on n values
     of u at x(1), x(2), ... x(n).

     If U is an array containing u(x) at these n points, then
     c.dot(U) will give the approximation to u^{(k)}(xbar).

     Note for k=0 this can be used to evaluate the interpolating polynomial
     itself.

    Requires len(x) > k.
    Usually the elements x(i) are monotonically increasing
    and x(1) <= xbar <= x(n), but neither condition is required.
    The x values need not be equally spaced but must be distinct.

    Modified rom  http://www.amath.washington.edu/~rjl/fdmbook/  (2007)
    """

    from  scipy.special import factorial

    n = x.shape[0]
    assert  k < n, " The order of the derivative must be less than the stencil width"

    # Generate the Vandermonde matrix from the Taylor series
    A = numpy.ones((n,n))
    xrow = (x - xbar)  # displacements x-xbar
    for i in range(1,n):
        A[i,:] = (xrow**(i))/factorial(i);

    b = numpy.zeros(n)    # b is right hand side,
    b[k] = 1              # so k'th derivative term remains

    c = numpy.linalg.solve(A,b)          # solve n by n system for coefficients

    return c

```

```python
N = 11
x = numpy.linspace(-2*numpy.pi, 2.*numpy.pi,  )
k = 2
scale = (x[1]-x[0])**k

print(fdcoeffV(k,x[0],x[:3])*scale)
for j in range(k,N-1):
    print(fdcoeffV(k,  x[j], x[j-1:j+2])*scale)
print(fdcoeffV(k,x[-1],x[-3:])*scale)
```

### Example: A variably spaced mesh

```python
N = 21
y = numpy.linspace(-.95, .95,N)
x = numpy.arctanh(y)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x,numpy.zeros(x.shape),'bo-')
axes.plot(x,y,'ro-')
axes.grid()
axes.set_xlabel('$x$')
axes.set_ylabel('$y$')
plt.show()
```

```python
k=1
fd = fdcoeffV(k,x[0],x[:3])
print('{}, sum={}'.format(fd,fd.sum()))
for j in range(1,N-1):
    fd = fdcoeffV(k,  x[j], x[j-1:j+2])
    print('{}, sum={}'.format(fd,fd.sum()))
fd = fdcoeffV(k,x[-1],x[-3:])
print('{}, sum={}'.format(fd,fd.sum()))

```

### Application to Numerical PDE's

Given an efficent way to generate Finite Difference Coefficients these coefficients can be stored in a (usually sparse) matrix $D_k$ such that  given any discrete vector $\mathbf{f} = f(\mathbf{x})$,  We can calculate the approximate $k$th derivative as simply the matrix vector product

$$
    \mathbf{f}' = D_k\mathbf{f}
$$

This technique will become extremely useful when solving basic finite difference approximations to differential equations (as we will explore in future lectures and homeworks).

### The Bigger idea

More generally, using finite differences we can transform a continuous differential operator on a function space

$$
    v = \frac{d}{dx} u(x)
$$
which maps a function to a function,  to a discrete linear algebraic problem

$$
    \mathbf{v} = D\mathbf{u}
$$
where $\mathbf{v}, \mathbf{u}$ are discrete approximations to the continous functions $v,u$ and $D$ is a discrete differential operator (Matrix) which maps a vector to a vector.


```python
from sympy import *
import matplotlib.pyplot as plt
from sympy.plotting import plot
from sympy.matrices import *
init_printing()
%matplotlib inline
```

# Problem 1 #

We have the following system

\begin{align}
    q_D &= b_D - 2p + r \\
    q_S &= b_S + 2p - r \\
    r &= 2b_r + 4p - 2q
\end{align}
Convert the system of equations into a linear function of the form $A\mathbf{x} = \mathbf{b}$

\begin{align}
    q + 2p - r &= b_D \\
    1q - 2p + r &= b_S \\
    1q - 2p + \frac{1}{2}r &= b_r
\end{align}

and in matrix terms we have

\begin{align}
    \begin{bmatrix}
        1 & 2 & -1 \\
        1 & -2 & 1 \\
        1 & -2 & \frac{1}{2}
    \end{bmatrix}
    \begin{bmatrix}
        q \\
        p \\
        r
    \end{bmatrix} &=
    \begin{bmatrix}
        b_D \\
        b_S \\
        b_r
    \end{bmatrix}
\end{align}

### Test Solution Existence and Uniqueness with Determinant
We want to test the determinant $\det A = 0$.  If it is, then we may either have no solution to the system, or an infinite number of solutions.  If $\det A \neq 0$ then we know for every vector $\mathbf{b}$ there is a unique solution to the system.

```python
A = Matrix([
        [1, 2, -1],
        [1, -2, 1],
        [1, -2, Rational(1,2)]
    ])

# We calculate the determinant using A.det() function
detA = A.det()
A, Eq(Symbol('\det A'),detA)
```

### Find A^{-1} and Solve the System

If $A$ is invertible then $\mathbf{x} = A^{-1}\mathbf{b}$.  So calculate the inverse and post multiply $\mathbf{b}$ to find the solution.  Assume that $\mathbf{b} = [10,5,1]$.

```python
#Instantiate b vector
b = Matrix([
        [10],
        [8],
        [2]
    ])

#Calculate the inverse of the A matrix.
Ainv = A.inv()
Ainv, b
```

```python
#Use the inverse of A to find the solution x
x = Ainv*b
x
```

# Problem 2

Consider the system:

\begin{align}
    q_D &= \frac{b_D}{p^2} \\
    q_S &= A_S + \frac{1}{b_S}p^2
\end{align}

```python
bD = Symbol('b_D', real=True, positive=True)
bS = Symbol('b_S', real=True, positive=True)
AS = Symbol('A_S', real=True, positive=True)
AD = Symbol('A_D', real=True, positive=True)
p = Symbol('p', real=True, positive=True)

demand = bD*exp(-p)
supply = bS*exp(p)

demand, supply
```

```python
#Find the equilibrium price
p_eq = solve(demand-supply,p)[0]
p_eq
```

```python
grad = Matrix([p_eq.diff(bD), p_eq.diff(bS)])
grad
```

```python
sl = [(bD,5),(bS,1)]
gradval = grad.subs(sl)
gradval
```

```python
dx = Matrix([Rational(1,9), -Rational(1,7)])
dp = gradval.dot(dx)
dp
```

```python
p_eq.subs(sl) + dp
```

```python
plot(5*exp(-p),1*exp(p),(p,0,2))
```

```python

```