## Topics covered in this notebook:
1. What is Linear Regression.
2. Simple 1D Linear Regression.
3. Quality of Fit - R-Squared.
4. Multiple Linear Regression.
5. Probabilistic interpretation of the squared error.
6. L2 Regularization or Ridge Regression.
7. Gradient Descent.
8. L1 Regularization or Lasso.
9. L1 vs. L2 Regularization.

## 1. Linear Regression

Consider a set of questions:
1. How to predict future stock price?
2. How to predict a rating of a movie?
3. How many followers will I get on twitter?
4. How can you predict a price of a house?

What does all these questions have in common?
1. **Outputs**. More specifically continous outputs (price of a house, # of followers etc.). Let's call this **'Y'** or **'y'**
2. Predicting continuous outputs is called **regression**

What do I need to predict outputs?
1. **Features** or Inputs, Let's call this **'X'**.
2. **Training examples**, many X's for which Y's are known.
3. A **model**, a function that represents the relationship between X & Y.
4. A **loss** or **cost** or **objective** function that tells us how our model represents the training examples.
5. **Optimization**, a way we can find the model parameters that minimizes the cost function.

## 2. Simple 1D Linear Regression

So let's re-state our problem in more general terms:
1. We are given a set of points: {(x1,y1),(x2,y2),....(xn,yn)}
2. We plot them in a 2D chart.
3. We find the line of best fit.

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
X = np.linspace(0,10,50)
Y = 2*X + np.random.normal(0,1,50)
```

```python
plt.scatter(X,Y,c = 'black',label = 'Data')
plt.plot(X,2*X,c = 'red',label = 'line of best fit')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend()
plt.show()
```

Our line of best fit is defined as:
\begin{equation}
 \large \hat y_i = ax_i+ b
\end{equation}

This is our **model**.

What can we do to make sure $\large y_i$ is closer to $\large \hat y_i$

Can we do this?

\begin{equation}
\large Error = \Sigma (\large y_i - \large \hat y_i)
\end{equation}

No. If the error for a point is -5 and +5 for the other the overall error is zero. However clearly that is not the case.

What we want is this:
1. For any target != predictions, a +ve contribution to the error.
2. Standard way is to square of difference.
3. Called the "sum of the squared errors".

\begin{equation}
\large Error (E) = \Sigma (\large y_i - \large \hat y_i)^2
\end{equation}

Now that we have error we have to minimize it. How do we do that?
 Take derivative & set it to zero!

Substitute equation (1) in (3):

\begin{equation}
\large E = \Sigma (\large y_i - \large ax_i- b)^2
\end{equation}

We need to minimize E w.r.t a & b. Hence we need to take partial derivative.

Derivative of above equation w.r.t a & b:

\begin{align}
\large \frac{\partial E}{\partial a} &= \Sigma \,2\, (\large y_i - \large ax_i- b)(-x_i) \\
\large \frac{\partial E}{\partial b} &= \Sigma \,2\, (\large y_i - \large ax_i- b)
\end{align}

Set that to zero:

\begin{align}
\large a\,\Sigma \large x_i^2 + \large b\,\Sigma \large x_i &= \Sigma \,\large x_i\,\large y_i \\
\large a\,\Sigma \large x_i + \large b\,\large N &= \Sigma \,\large y_i \\
\end{align}

Solve those 2 equations simultaneously:

\begin{align}
\large a\, &= \frac{N\,\large\Sigma \,\large x_i\,\large y_i - \Sigma \,\large x_i\,\Sigma \,\large y_i}{N\, \large\Sigma \large x_i^2 - (\Sigma \large x_i)^2} \\
\large b\, &= \frac{N\,\large\Sigma \,\large x_i\,\large y_i^2 - \Sigma \,\large x_i\,\Sigma \,\large x_i \large y_i}{N\, \large\Sigma \large x_i^2 - (\Sigma \large x_i)^2} \\
\end{align}

Simplify it further is diving the above equations by $N^2$ and utilize the defitions of mean:
\begin{align}
\large a\, &= \frac{\overline{\large xy} - \overline{\large x}\,\overline{\large y}}{\overline{\large x^2} - \overline{\large x}^2}\\
\\
\large b\, &= \frac{\overline{\large y}\, \overline{\large x^2}  - \overline{\large x}\,\overline{\large xy}}{\overline{\large x^2} - \overline{\large x}^2}\\
\end{align}

where,

\begin{align}
\large \overline{x}\, &= \frac{1}{N} \,\large \Sigma \,\large x_i \\
\large \overline{xy}\, &= \frac{1}{N} \,\large \Sigma \,\large x_i\,y_i \\
\end{align}

## 3. Quality of Fit: R-Sq

\begin{equation}
\large R^2 \, = 1 \, - \frac{SS_{\text{res}}}{SS_{\text{tot}}} \\
\end{equation}

where
\begin{align}
\large SS_{\text{res}}\, &=  \large \Sigma (y\, - \hat y_i)^2 \\
\large SS_{\text{tot}}\, &=  \large \Sigma (y\, - \overline y_i)^2 \\
\end{align}

1. Suppose the residual (SS_res) is close to zero. Then the R-Sq will be ~1 -> Perfect correlation.
2. Suppose the R-Sq is zero which means SS_res / SS_tot = 1 -> This means we predicted just the everage of y. Model not good.
3. When is R-Sq < 0? This means SS_res / SS_tot > 1. Model prediction is worse than mean. Again model not good.

## 4. Multiple Linear Regression

The error function still doesn't change from 1D regression, only the expression for the prediction does,

\begin{equation}
\large Error (E) = \Sigma (\large y_i - \large \hat y_i)^2 = \Sigma (\large y_i - \large w^Tx_i)^2
\end{equation}

We can still take derivative w.r.t to any component of w: j = 1,....D(No. of inputs or feature dimension)

\begin{align}
\large \frac{\partial E}{\partial w_j} &= \Sigma 2(\large y_i - \large w^Tx_i)(- \frac{\partial (w^Tx_i)}{\partial w_j})\\
&= \large \Sigma \,2(\large y_i - \large w^Tx_i)(- x_{ij})
\end{align}

Set this equation to zero. D equations & D unknowns:

\begin{align}
\large \Sigma \,2(\large y_i - \large w^Tx_i)(- x_{ij}) &= 0\\
\large \Sigma \,w^Tx_ix_{ij} - \large \Sigma y_ix_{ij} &= 0
\end{align}

Isolate w & represent everything in matrix form using dot product:

\begin{align}
\large w^T(X^TX)\, = \, y^TX\\
\end{align}

Take transpose on both sides:

\begin{align}
\large [w^T(X^TX)]^T\, &= \, [y^TX]^T\\
\large (X^TX)w\, &= \, X^T\large y\\
\large w \, &=\, (X^TX)^{-1}\, X^T\large y
\end{align}

Use numpy linalg.solve to solve the above equation. Elimiates the need to manually taking inverse:

\begin{align}
\large w \, &=\, np.lingalg.solve(X^TX\,, X^T\large y)
\end{align}

## 5. Probabilistic Interpretation of Squared Error:

### Linear Regression is the maximum likelihood solution to the line of best fit.

What is Maximum Likelihood?
1. Assuming Gaussian distribution - If we plot histogram for grades of students we will get a bell curve:

2. We would do an experiment to collect everyone's grades: {x1, x2,....xn}

3. Intuitively, we know the average grade of the students is:
\begin{align}
\large \mu \, &= \frac{1}{N} \,\large \Sigma \,\large x_i \\
\end{align}
4. Is there a systematic way of getting this answer? Suppose we want to find thr true mean of the Gaussian form which the data arises:

\begin{align}
 find\, \large \mu \,,\, where\,X \, \tilde \, N(\mu , \sigma^2) \\
\end{align}
5. We can write the probability of any single point xi:

\begin{align}
\large p(x_i) \, &= \frac{1}{\sqrt{2\pi\sigma^2}} \,\large exp^{\frac {1}{2} \, \large \frac {(\large x_i-\ \large mu)^ 2}{\large \sigma^2}}\\
\end{align}
6. Since we know the grades are iid, we can muliply individual probabiities:
\begin{align}
\large p(x_1,x_2,...x_N) \, &= \prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}} \,\large exp^{\frac {1}{2} \, \large \frac {(\large x_i-\ \large mu)^ 2}{\large \sigma^2}}\\
\end{align}
7. Another way to write this is as 'likelihood' form. Probability of X given the parameter of interest:
   1. We want to find mean so that likelihood is maximized. This is called 'maximum likelihood'.
   2. We want to finfd the best setting of mean(mu) so that the date we measured is likely to have come from this distribution.
\begin{align}
\large p(X\,|\,\mu) = \large p(x_1,x_2,...x_N) \, &= \prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}} \,\large exp^{\frac {1}{2} \, \large \frac {(\large x_i-\ \large mu)^ 2}{\large \sigma^2}}\\
\end{align}
8. How to find mu? Take log of the likelihood & set it to zero to make it easy to solve. log() in monotomically increasing function so if A > B -> log(A) > log(B)
\begin{align}
\large l \, &= \large log \prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}} \,\large exp^{\frac {1}{2} \, \large \frac {(\large x_i-\ \large mu)^ 2}{\large \sigma^2}}\\
\large l \, &= \large [\, \Sigma_{i=1}^n-\frac{1}{2} log(\sqrt{2\pi\sigma^2}) \,- \large {\frac {1}{2} \, \large \frac {(\large x_i-\ \large mu)^ 2}{\large \sigma^2}}\,]\\
\end{align}
Take derivative and set it to zero:
\begin{align}
\large \frac{dl}{d\mu} &= \large \Sigma \, \frac {(x_i - \mu)}{\sigma^2} = 0\\
\large \mu \, &= \frac{1}{N} \,\large \Sigma \,\large x_i \\
\end{align}

9. If we look at the equation for l and remove some irrelevant terms that are absorbed by zero, we get:
\begin{align}
\large l \, &= \large [\, \Sigma_{i=1}^n-\frac{1}{2} log(\sqrt{2\pi\sigma^2}) \,- \large {\frac {1}{2} \, \large \frac {(\large x_i-\ \large mu)^ 2}{\large \sigma^2}}\,]\\
\large equivalent \,l &= \large - \Sigma \, (x_i - \mu)^2\\
\end{align}

### This means maximizing the log-likelihood is equivalent to minimizing negative of squared errors
Compare this to our error(E):
\begin{align}
\large \,l &= \large - \Sigma \, (x_i - \mu)^2 -> \,maximize\\
\large E & = \large \Sigma\,(\large y_i - \large \hat y_i)^2 -> \, minimize \\
\end{align}
So minimizing E is equivalent to maximizing -E. So when we minize the squared error for linear regression, this is equivalent to maximizing the likelihood.

Equivalent way of writing this:
\begin{align}
\large y \,\tilde \,\, N(w^Tx,\sigma^2)\\
\end{align}

\begin{align}
\large y & = \, \large w^Tx\, +\, \epsilon,\, \epsilon\, \tilde\, N(0,\sigma^2) \\
\end{align}

### In other words, linear regression makes the assumption that errors are gaussion and the trend is linear.

## 6. L2 Regularization or Ridge Regression:

Data may have outliers. These outliers are usually measurement errors or data entry errors. In some cases these outliers may actually mean something but in this case lets assume its the former assumption.
#### Outliers pull the line of best fit away from the main trend to minimize the squared error. The idea is to ensure the weights are not overly large weights because that might want to fit to outliers.
Look at the 2 lines below. Which one is the best fit? Red is the standard model which accounts for the outliers whereas black is the L2 regularization model that ignored the effect of outliers.

### How does L2 Regularization work?
Modify the cost function (J) such that large weights are penalized.

\begin{align}
\large J & = \large \Sigma\,(\large y_i - \large \hat y_i)^2 + \lambda \lvert \large w^2\rvert \\
\lvert \large w^2\rvert &= \large w^Tw = \large w1^2 + w2^2 + ....w_D^2
\end{align}

#### Probabilistic Perspective:
1. Plain squared error maximizes likelihood because J = negative log likelihood.
2. Now we are no longer maximizing this, since there are 2 terms. The 2nd is called prior which has the information about the weights irrespective of the data since it is not dependent on x & y.

3. Looks like Bayes rule.
#### This is called MAP - maximum a posteriori.

4. Solve for w:

## 7. Gradient Descent:

1. Optimization method.
2. Used extensively in deep learning.
3. Idea: You have a fucntion you want to minimize, J(w) = cost or error. Find optimal inputs to minimize this function.

4. Weight, w = w - learning_rate * dJ/dw.
5. Example: J = w^2:
   1. dJ/dw = 2w.
   2. Set initial  w = 10, learning rate = 0.1.
   3. Iteration 1: w = 10 - 0.1 * 20 = 8.
   4. Iteration 2: w = 08 - 0.1 * 16 = 6.4.
   5. Iteration 3: w = 6.4 - 0.1 * 12.8 = 5.12 and so on..
   6. Notice how the weigh 'w' is converging towards the true solution 0. With enough iterations we will coverge.

```python
import numpy as np
```

```python
w = 5
learning_rate = 0.1
```

```python
for i in range(50):
    w -= learning_rate * 2* w
    print(w)
print('Notice how w is converging towards zero!')
```

    4.0
    3.2
    2.56
    2.048
    1.6384
    1.31072
    1.0485760000000002
    0.8388608000000002
    0.6710886400000001
    0.5368709120000001
    0.4294967296000001
    0.3435973836800001
    0.27487790694400005
    0.21990232555520003
    0.17592186044416003
    0.140737488355328
    0.11258999068426241
    0.09007199254740993
    0.07205759403792794
    0.057646075230342354
    0.04611686018427388
    0.03689348814741911
    0.029514790517935284
    0.02361183241434823
    0.018889465931478583
    0.015111572745182867
    0.012089258196146294
    0.009671406556917036
    0.007737125245533628
    0.006189700196426903
    0.004951760157141522
    0.003961408125713218
    0.0031691265005705745
    0.00253530120045646
    0.0020282409603651678
    0.0016225927682921343
    0.0012980742146337075
    0.001038459371706966
    0.0008307674973655728
    0.0006646139978924582
    0.0005316911983139665
    0.00042535295865117324
    0.0003402823669209386
    0.00027222589353675085
    0.0002177807148294007
    0.00017422457186352054
    0.00013937965749081642
    0.00011150372599265314
    8.920298079412252e-05
    7.136238463529802e-05
    Notice how w is converging towards zero!

### Gradient Descent for Linear Regression:

Cost function to minimize:
\begin{align}
\large J \, &= \large (Y\, - Xw)^T (Y\, - Xw) \\
\end{align}

Gradient:
\begin{align}
\large \frac {\partial J}{\partial w} \, &= \large -2X^TY\, +\, 2X^TXw \,=\, 2X^T(\hat Y\, -\, Y) \\
\end{align}

#### Instead of setting it to 0 & solving for w, we will just take small steps in this direction.

So weight update will be for a certain number of steps:

\begin{align}
\large w \, &= \large w - \eta * X^T(\hat Y\, -\, Y) \\
\end{align}

## 7. L1 Regularization (Lasso):

In general we want no. of features(D) << no. of samples(N). Also, there might be a case where some of the features are just constant or quasi-const, uncorrelated to output etc. In all these cases we could use L1 regularization to eliminate those features from the model.

1. Select a small number of important features that actually predict the trend.
2. Eliminate the noise influence on the output.
3. Similar to L2. It has a penalty term using L1 norm.

4. Similar to L2 this also puts a prior on w, so its also a MAP estimation of w. We had a gaussian distribution on the prior for L2. We have Laplace here:
\begin{align}
\large p(w) \, &= \large \frac {\lambda}{2} exp(- \lambda \lvert \large w\rvert) \\
\end{align}
5. Taking derivative of cost function:
\begin{align}
\large \frac {\partial J}{\partial w} \, &= \large -2X^TY\, +\, 2X^TXw \,+\, \lambda sign(w) \\
\end{align}
6. Since this has a sign term we can't used closed form solution to update weights and use gradient descent instead.

## 9. L1 vs L2 Regularization

L2 reduces the effect of outliers on the model by maintaining the weights to a smaller value.
L1 removes unwanted features. Encourages a sparse solution.

Both helps prevent overfitting, by not fitting to noise.

L2 penalty is quadratic: as w -> 0 derivative ->0

L1 penalty is abs func : as w -> 0 doesn't matter it will fall at a contant rate. When it reaches 0, is stays there forever.

#### Combine L1 & L2 = ElasticNet

## References:
1. An Introduction to Statistical Learning Textbook by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.
2. University of Michigan EECS 445 - Machine Learning Course (https://github.com/eecs445-f16/umich-eecs445-f16).

3. University of Toronto CSC 411 - Intro. to Machine Learning (http://www.cs.toronto.edu/~urtasun/courses/CSC411_Fall16/CSC411_Fall16.html).

4. Stanford CS109 - Intro. to proabability for computer scientists (https://web.stanford.edu/class/archive/cs/cs109/cs109.1166/).

5. Few online courses on Udemy, Coursera etc.


## Define dcreasing regimes for N' and N'' (belly and no-belly)

$C(z)=C_0 + C'(z-z_0) + C''(z-z_0)^2$, where $N_0$ is the concentration at $z_0$. I will start with $z_0=H_s$, so $C_0$ is a reference concentration at shelf break depth.

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sympy as sym
import seaborn as sns
sym.init_printing() # enable fancy printing
```

    /Users/Karina/anaconda3/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
      warnings.warn(self.msg_depr % (key, alt_key))

```python
# Set appearance options seaborn
sns.set_style('white')
sns.set_context('notebook')
```

```python
C1,C2,z,Co,Hs,tau,Z = sym.symbols('C1,C2,z,Co,Hs,tau, Z')
func = Co + C1*(z-Hs) + C2*((z-Hs)**2)

```

```python
func
```

### What should be the value of $\hat{C''}$ to avoid maxima, minima or inflectio points within Hd?

i.e. Profiles with 'no bellies' (max, mins or inflection points)

There are bellies wherever

$d\hat{C}/dz=\hat{C'}+2\hat{C''}(z-Hs)=0$. (1)

I don't want these points (I'll call them $z_{belly}$) to be within my profile ($0\le z \le Hd$), so I have the condition:

  $0 \ge z_{belly}$ or $z_belly \le 400$ (2),

but solving (1) for $z_{belly}$ gives,

$z_{belly}=\frac{-\hat{C'}}{2\hat{C''}}+Hs$ (3).

The inequalities (2) and (3) give:

(4) $\frac{\hat{C'}}{2Hs}\ge {\hat{C''}}$ and (5) $\frac{\hat{C'}}{2(Hs-Hd)}\le {\hat{C''}}$,

for a given $ 0>\hat{C'}$ (since I want decreasing profiles)

The region that satisfies conditions (4) and (5) is the triangle plotted below. Also, these hold for the dimensional parameters $C'$ and $C''$, where $  0 > C'$

```python
p2 = sym.plot_implicit(sym.Or(C1/(2*(147.5-400)) < C2,C1/(2*147.5) > C2),(C1,-0.5,0),(C2,-0.002,0.002),
                      title='C2 values as a function of C1 values, no-belly')

```

```python
p2 = sym.plot_implicit(sym.And(C1/(2*(147.5-400)) > C2,C1/(2*147.5) < C2),(C1,-0.5,0),(C2,-0.002,0.002),
                      title='C2 values as a function of C1 values, belly regime')

```

Let's see if it works...

```python
func = 0 + C1*(z-Hs) + C2*((z-Hs)**2)
hand = sym.plot(func.subs({C1:-0.00005,Hs:147.5,C2:0.0005/(2*147.5)}),
               func.subs({C1:-0.003,Hs:147.5,C2:0.03/(2*147.5)}),
               func.subs({C1:-0.005,Hs:147.5,C2:-0.005/(2*147.5)}),
               func.subs({C1:-0.01,Hs:147.5,C2:-0.0067/(2*147.5)}),(z, 0, 400),
               xlabel='Depth (m)',
               ylabel='Concentration (nd)',
               title='Using the upper bound of C2',
               show=False)
hand[1].line_color='r'
hand[2].line_color='g'
hand[3].line_color='purple'

hand.show()
```

## $\tau_v$ as a function of N' and N''

$$\frac{\tau}{Z}=\frac{\delta^2_vN}{\delta_vN}=Z\frac{\delta^2_vN}{\delta_vN}\rvert_{z=Hs}=\frac{2N''}{N'}$$

We have some bounds for N' and N'': we imposed $0\le N'$ and $N'/2Hs\gt N''$ and $N''\lt N'/(2(Hs-400))$ to have a profile with a max, min or inflection point above 400 m.

Using these bounds, we can now bound the values of \tau_v/Z:

$$\frac{\tau}{Z}=\frac{2N''}{N'} \gt \frac{2N'}{2HsN'} = \frac{1}{Hs}$$

and

$$\frac{\tau}{Z}=\frac{2N''}{N'} \lt \frac{2N'}{2N'(Hs-Hd)} = \frac{1}{(Hs-Hd)}$$

These bounds do not depend on N' or N'', only on Hs.

So,

$$ \frac{1}{Hs-Hd} \gt \frac{\tau}{Z} \gt \frac{1}{Hs}$$

Hs=0 and Hd are asymptotes, but that is fine. If Hs = 0 it means you don't even have a shelf and if Hs=400 m you probbaly care about not having bellies deeper than that, so 400 m wouldn't even be an asymptote in the first place.

```python
p2 = sym.plot_implicit(sym.Or(1/Hs < tau, 1/(Hs-400)> tau), (Hs, 0, 400),(tau,-0.1,0.1),
                      title='Tau/Z as a function of Hs',
                      ylabel='Tau/Z')

```

In the idealized bathymetry Hs=147.5 m, so the bounds are: $-0.004  \gt \frac{\tau}{Z} or  \frac{\tau}{Z}\gt 0.007$, units are m$^{-1}$. If I say that Z can be between 0 and 100 m then the values of $\tau_v$ can be:

```python
p2 = sym.plot_implicit(sym.Or(Z/147.5 < tau, Z/(147.5-400)> tau), (Z, 0, 100),(tau,-0.5,0.7),
                      title='Tau as a function of Z',
                      ylabel='Tau')

```


# Final review

## 1- Measurement systems
### Goal of experiments
- __1- Engineering/scientific experimentation__
- __2- Operational system__

### Units - Unit conversion - Dimensional analysis
### Errors and Uncertainties

## 2- Dynamic response of measurement systems

\begin{align*}
a_n\frac{d^ny}{dt^n} + a_{n-1}\frac{d^{n-1}y}{dt^{n-1}}x + \dots + a_1 \frac{dy}{dt} + a_0 y = F(t)
\end{align*}

###  0$^{\mathrm{th}}$ order system

\begin{align*}
y(t) = K F(t) = \frac{1}{a_0} F(t) \quad \text{$K=1/a_0$ is called the _static sensitivity_ or _steady-state gain_}
\end{align*}

###  1$^{\mathrm{st}}$ order system

\begin{align*}
a_1 \frac{dy}{dt} + a_0 y = F(t) \quad
\tau \frac{dy}{dt} + y = K F(t)
\end{align*}
with $\tau = a_1/a_0$ the time constant of the system.

###  2$^{\mathrm{nd}}$ order system

\begin{align*}
a_2 \frac{d^2y}{dt^2} + a_1 \frac{dy}{dt} + a_0 y = F(t) \quad
\frac{1}{\omega^2_n} \frac{d^2y}{dt^2} + \frac{2\zeta}{\omega_n} \frac{dy}{dt} + y = K F(t)
\end{align*}

## 3- Digital data acquisition
### Binary to decimal conversion

### ADC system
__Resolution__
\begin{align*}
\Delta V = \frac{V_{max} - V_{min}}{2^N}
\end{align*}

__Quantization error__
\begin{align*}
Q = \pm \frac{1}{2} \Delta V = \pm \frac{V_{max} - V_{min}}{2^{N+1}}
\end{align*}

__Clipping__
### Discrete Sampling

### Aliasing
Nyquist Shannon sampling rate thereom states:
>_If a function, $y(t)$, contains no frequencies greater than $f_{signal}(MAX)$, it is completely determined if it is sampled with a sampling frequency $f_s$ such that:_
\begin{align*}
f_s > 2 f_{signal}(MAX)
\end{align*}
> if $f_s \ngtr 2f_{signal}(MAX)$
\begin{align*}
f_a =  \left| f - f_s \cdot \mathrm{NINT}\left( \frac{f}{f_s} \right) \right|
\end{align*}

### Fourier Series
\begin{align*}
y(t) = \frac{A_0}{2} + \sum_{n=1}^{\infty} \left( A_n \cos \left(\frac{2\pi nt}{T}\right) + B_n \sin \left(\frac{2\pi nt}{T}\right) \right)
\end{align*}
- __Fundamental harmonic/frequency__: $n=1$, noted $\omega_0=2\pi/T$ or $f_0 = 1/T$.
- $T$: signal period
- Frequencies of $\sin$ and $\cos$ are multiple of the fundamental frequency $\omega$: __$n^{th}$ harmonic__.
- $A_n$, $B_n$: Fourier coefficients
- $A_0/2$: mean of the function

### Fourier Transform, DFT, FFT
\begin{align*}
Y(k \Delta f) = \sum_{n=0}^{N-1} y(n \Delta t) e^{-i (2\pi k \Delta f)(n\Delta t)} \qquad k = 0,\,1,\,2, \dots , \, N-1
\end{align*}

\begin{array}{ll}
N & \text{number of samples taken over sampling interval $T_s$}\\
T_s & T_s = N\Delta t\quad\text{total sampling time}\\
\Delta f & \text{frequency increment/resolution}\\
 & \Delta f = 1/T_s = 1/N\Delta t\\
\Delta t & \text{time increment (or sampling period)}\\
 & \Delta t = T_s/N\\
f_s & \text{sampling rate}\\
 & f_s = 1/\Delta t = N/T_s\\
f_{max} & f_{max} = f_{folding} = f_s/2 = N/2 \cdot \Delta f \quad \text{max resolved frequency}
\end{array}

### Leakage
### Windowing
### Anti-aliasing filter

### Analog signal reconstruction
If an analog signal is sampled properly: reconstruct the analog signal with the _cardinal series_.

## 4- Signal Conditioning

### Basic electronics

### Filters
#### $1^{\mathrm{st}}$ order low-pass filter

\begin{align*}
\omega_{cutoff} & = \frac{1}{RC} \quad  f_{cutoff} =  \frac{\omega_{cutoff}}{2 \pi} = \frac{1}{2\pi RC} \\
G & = \frac{|V_{out}|}{|V_{in}|} =  \frac{1}{\sqrt{1+ \left( \frac{f}{f_{cutoff}}\right)^2 }} \quad
\phi = -\arctan \left( \frac{f}{f_{cutoff}} \right)
\end{align*}
#### Low-pass filter of order $n$
\begin{align*}
G = \frac{1}{\sqrt{1+ \left( \frac{f}{f_{cutoff}}\right)^{2n} }}
\end{align*}
#### $1^{\mathrm{st}}$ order high-pass filter
\begin{align*}
G = \frac{1}{\sqrt{1+ \left( \frac{f_{cutoff}}{f}\right)^2 }} \quad
\phi = \arctan \left( \frac{f_{cutoff}}{f} \right)
\end{align*}

#### High-pass filter of order $n$
\begin{align*}
G = \frac{1}{\sqrt{1+ \left( \frac{f_{cutoff}}{f}\right)^{2n} }}
\end{align*}

### Op-Amps
ideal op-amps vs real effects (impedance loading, GBP, CMRR)

## 5- Design Stage uncertainty analysis
### Expected uncertainty
RSS: root of the sum of the squares
### Uncertainty of a result
\begin{align}
R = f(x_1, x_2, \dots) \quad u_R = \sqrt{\sum_i \left( u_{x_i} \frac{\partial f}{\partial x_i} \right)^2}
\end{align}
If f is polynomial
\begin{align}
f = x_1^{a_1} \cdot x_2^{a_2} \cdot \dots \quad \frac{u_R}{R} = \sqrt{\sum_i \left( \frac{u_{x_i}}{x_i} a_i \right)^2}
\end{align}
### Combining elemental uncertainties
\begin{align}
u_{sys} = \sqrt{\sum_i u_i^2} \quad \text{with $u_i$ elemental error}
\end{align}

# 6- Design of experiments
## Main concerns of DOE
- Validity of results
- Reliability
- Reproducibility
- Statistical significance
- Define the problem

## Design steps
- Design the experiment
- Construct the experiment
- Acquire data
- Analyze data
- Do confirmatory experiments (if necessary)
- Interpret and report results and methodology.

## Test matrix design

### Full factorial analysis

$P$: number of parameters,
$L$: number of levels each parameter is tested at,
$ N= L^P$ tests.
### Fractional factorial analysis
### Taguchi design array

Optimum Taguchi Design Array for fractional factorial analysis
- Each level of each parameter appears the same number of times in the array.
- Repetitions of parameter-level combinations are minimized as much as possible.

# 7- Strain Gage
\begin{align*}
\sigma_a = \frac{F}{A}, \qquad \epsilon_a = \frac{dL}{L}, \qquad \sigma_a = E \epsilon_a
\end{align*}

\begin{align*}
\frac{dR}{R} = S \epsilon_a
\end{align*}

## Wheatstone bridge

\begin{align*}
\frac{V_o}{V_s} = \frac{1}{4} \left( \frac{dR_1}{R_{ini}} - \frac{dR_2}{R_{ini}} + \frac{dR_3}{R_{ini}} - \frac{dR_4}{R_{ini}} \right)
\end{align*}

### Bridge of order $n$
\begin{align*}
\frac{V_o}{V_s} = \pm \frac{n}{4} \epsilon_a S
\end{align*}
quarter-bridge: $n=1$, half-bridge: $n=2$, full-bridge: $n=4$.

# 9- Pressure Measurement
\begin{align}
P_{gage} = P_{abs}-P_{atm}
\end{align}

\begin{align}
\left| \frac{P}{P_0} \right| = \frac{1}{\sqrt{\left( 1 - \left( \frac{f}{f_n} \right)^2  \right)^2 + 4h^2 \left( \frac{f}{f_n} \right)^2 }}
\end{align}
with $f_n$ natural frequency, $h$ damping ratio
\begin{align}
f_n = \sqrt{\frac{3 \pi r^2 c^2}{4L V}} \quad h = \frac{2 \mu}{\rho c r^3} \sqrt{3LV}{\pi}
\end{align}
- Mechanical

Manometer, Bourdon gages

- Electronic

Displacement, deflection, piezo

# 10- Temperature Measurement

## Mechanical
## Thermoresitive
### Resistance Temperature Device (RTD):
Same than strain gage: $R(T) =R_0 + a (T-T_0)$
### Thermistor

Semi-conductor material: $ R(T) = R_0 \exp \left( \beta \left( \frac{1}{T} - \frac{1}{T_0} \right) \right) $

## Thermoelectric (thermocouple-TC)

Seebeck effet: emf(T) when connecting dissimilar conductors
- Law of intermediate metals
- Law of intermediate temperatures

## Radiative

# Sample Problems
## Quiz
> What are the major elements of a measurement system?

> When acquiring a repeatable signal of fixed (and unknown) frequency with a digital data acquisition system, how can you determine experimentally if you have aliasing in your data?

> Are there any outliers in the time series data below?  Briefly explain.  On the top graph, the data are in blue, the curve fit in red.   The bottom graph is the residuals with the $95\%$ confidence interval in dashed lines.

> Give two tests to check if your data are randomly distributed.

> What is the benefit of using Taguchi array when designing a matrix of experiments?

### Force Balance
When subjected to a point load on its tip, a rectangular cantilever beam of base (width), $b$, and height (thickness), $h$ has axial strain, $\epsilon$, on the surface of its base:

\begin{align}
\epsilon = \frac{6 W L}{E b  h^2}
\end{align}

where $E$ is the Young's modulus, $W$ the load, and $L$ the distance from the free tip of the beam to the measurement point.

The following parameters are known about the beam:
It has on overall length of 1 m.  $h= 0.00500\, \text{m} \pm 0.4$%, $b = 0.0200\, \text{m}\pm 0.1$%, $E = 200 \,\text{GPa} \pm 0.05$%. Finally, the strain gage is mounted at $L = 0.5\text{ m} \pm 0.005 \text{ m}$.

You wish to measure the strain using a quarter-bride and one strain gage mounted on the surface of the top base at location $L$.  By convention, you would like to have a positive voltage output for a downward deflection of the beam.

>  Draw a Wheatstone bridge and show where the strain gage should be plugged.  Provide an explanation for your choice.

SG top, -> $\epsilon_a > 0$ for $W >0$

SG $R_1$ or $R_3$

> Predict the uncertainty on the force measurement, neglect the uncertainty on the axial strain.

Uncertainty of a result, $W = (L, E, b, h)$

W is polynomial
\begin{align}
\frac{u_W}{W} = \sqrt{\sum_i \left( \frac{u_{x_i}}{x_i} a_i \right)^2}
\end{align}

> Which dimension or parameter contributes the most to your uncertainty?

> How would you increase the resolution of your measurements on the applied force with this system, i.e. which parameter or dimension should you change or refine?

increase $L$: more sensitivity, smaller $u_L/L$.

> The beam is expected to see a lot of temperature variations and strain gages are sensitive to temperature.  How would you change your measurement system to compensate for temperature effects on your strain measurement?  Explain briefly why.

> Bonus question :
A two-component force balance is made of two cantilever beams (like the one above), with a beam dedicated to measuring the axial force while the other measures the side force.  In your tests, the axial force is 10 times larger than the side force.  For practical reasons, the beams are made of the same materials and with identical height and length.  How would you size the rest of your system to obtain the most sensitive two-component force balance.

## Signal conditioning and analysis
You are being tasked with designing a load cell to measure oscillations at 260 Hz.  You choose to use a quarter-Wheatstone bridge and size your load cell such that the strain is up to 500 $\mu$-strain.  You are being told that there will be electromagnetic noise in your signal.

You resulting design has the following parameters:
- There is noise on the input: $f_{noise}$ = 60 Hz with amplitude $\pm$ 0.50 mV .
- The strain factor is $S = 2.00$.
- The supply voltage for the Wheatstone bridge is $V_s = 5.00$ V.
- The output is sent to a DAQ board with 12-bit A/D converter and a range of $\pm 10$ V.
- All Wheatstone bridge resistors (and the strain gage) have resistance 350 $\Omega$ including the unloaded strain gage, resulting in a balanced bridge at zero strain.

> Propose a sampling frequency for your data acquisition system that would maximize your frequency resolution.  Briefly justify your choice.

> To make the best use of your digital data acquisition system, what should be the gain of your amplifier stage in your signal conditioning?  In practice would you accomplish this with one or two amplifier stages?  Briefly justify why.

```python
V_0 = 1/4*2.00*5E-4*5.00 # V
Q = (10--10)/2**(12+1)
print(V_0, Q)
```

    0.00125 0.00244140625

```python
G = 8/0.00125
print(G**(1/3))
```

    18.566355334451114

> Which type of amplifier stages will you use in your design (inverting or non-inverting)?  Explain why.

> Propose a scheme to filter the noise in your expected data.  In particular select the cutoff frequency(iess) and order of the filter.  Show all your steps.

> Bonus question: Before building your filter, you decide to acquire data to assess the signal and noise in your system.  You obtain the data and spectrum below.  What is your assessment of the actual carrier frequency(ies) of your signal and the nature of the noise?  Propose a new scheme (qualitatively) to treat your data.

```python

```


# Quadrotor Model

Multirotors are underactuated aerial robots that consist of multiple motor/propeller combinations. A quadrotor has four motor/propellor combinations, as shown in the below figure.

Figure

## Coordinate Frames

Coordinate frames are rigid, orthonormal bases of $\mathbb{R}^3$. There are two main coordinate frames of interest: the inertial frame $\mathcal{F}^i$ and the body frame $\mathcal{F}^b$. The inertial frame $\mathcal{F}^i$ is an orthonormal basis of $\mathbb{R}^3$ that is earth-fixed with an origin defined as the ''home'' position. The body frame $\mathcal{F}^b$ represents the quadrotor and we assume that its origin is the center of mass of the quadrotor.

Intermediate frames exist that help us account for the roll, pitch, and yaw rotations of the quadrotor. The first intermediate frame is the vehicle frame $\mathcal{F}^v$, which is at the position of the unrolled, unpitched, unyawed quadrotor. The vehicle frame leads to the vehicle-1 frame $\mathcal{F}^{v1}$ by yawing about the $\mathbf{k}^v$ axis, resulting in the unrolled, unpitched frame. Pitching about the new $\mathbf{j}^{v1}$ axis results in the unrolled vehicle-2 frame $\mathcal{F}^{v2}$, and rolling about the new $\mathbf{i}^{v2}$ axis results in the properly oriented body frame $\mathcal{F}^b$.

Additional frames may be the camera frame, the gimbal frame, or other sensor or measurement frames.

Note that in this work, we use the aerospace standard of passive, right-handed rotations and we will use 3-2-1 ($\psi$-$\theta$-$\phi$) Euler angles for yaw, pitch, roll, respectively.

Python code that builds proper rotation matrices in $\mathbb{R}^{3\times3}$ is given below. These rotation matrices can be used to rotate from the vehicle (or inertial) frame to the body frame through all the intermediate frames.

```python
import numpy as np

def Rot_v_to_v1(psi):
    R = np.array([
        [ np.cos(psi), np.sin(psi), 0],
        [-np.sin(psi), np.cos(psi), 0],
        [     0,           0    ,   1]
    ])
    return R

def Rot_v1_to_v2(theta):
    R = np.array([
        [np.cos(theta), 0, -np.sin(theta)],
        [      0      , 1,        0      ],
        [np.sin(theta), 0,  np.cos(theta)]
    ])
    return R

def Rot_v2_to_b(phi):
    R = np.array([
        [1,       0,           0     ],
        [0,  np.cos(phi), np.sin(phi)],
        [0, -np.sin(phi), np.cos(phi)]
    ])
    return R

def Rot_v_to_b(phi, theta, psi):
    return Rot_v2_to_b(phi).dot(Rot_v1_to_v2(theta).dot(Rot_v_to_v1(psi)))

def Rot_i_to_b(phi, theta, psi):
    return Rot_v_to_b(phi, theta, psi)
```

## Configuration Space

The quadrotor equations of motion consist of $6$ degrees of freedom (DOF) and $12$ ordinary differential equations ($6$ are kinematic and $6$ are dynamic). The quadrotor is modelled as a freely-rotating rigid body evolving in $\mathcal{M} = \mathbb{R}^3 \times \text{SO}(3) = \text{SE}(3)$.

Note that $\text{SO}(3)$ is a matrix Lie group that describes the orientation of rigid body in 3D space and is not isomorphic to $\mathbb{R}^3$. However, because Lie groups are smooth (i.e., differentiable) manifolds, there is an atlas of charts (i.e., homemorphisms) that allow the manifold to be locally described as a Euclidean space. A popular set of local parameterizations are the intuitive Euler angles, of which there are twelve. We will use the intrinsic 3-2-1 ($\psi$-$\theta$-$\phi$) Euler angle representation as is common in aerospace with first a yaw rotation about $\mathbf{k}$, a pitch rotation about $\mathbf{j}$, and then a roll rotation about $\mathbf{i}$. Additionally, we assume that the roll, pitch, and yaw of the quadrotor are small -- a logical assumption for the smooth trajectories of typical maneuvers. More precisely, we have that
$$
\phi \in [0, 2\pi) \subset \mathbb{R} \\
\theta \in (-\pi/2, \pi/2) \subset \mathbb{R} \\
\psi \in [0, 2\pi) \subset \mathbb{R}.
$$
Thus, the state of the quadrotor is given by $\mathbf{x} = \begin{bmatrix} x & y & z & \phi & \theta & \psi \end{bmatrix}^\top \in \mathbb{R}^6$.

## Equations of Motion

### Kinematics

Kinematics describe the motion of an object without consideration of what caused that motion. Using kinematics, we can relate position to velocity and orientation to angular velocity. Notice that these equations are irrespective of intertial properties such mass and size.

#### Translational

$$
\dot{\mathbf{r}}^i = R_b^i \mathbf{v}^b
$$

#### Rotational

When determining the rotational kinematic equation of a rigid body, we are asking how the angular velocities relate to the current orientation. A rate gyro is used to measure the angular velocity about each of the body-frame axes of the rigid body. These gyro measurements are given by $\mathbf{\omega}^b = p\mathbf{i}^b + q\mathbf{j}^b + r\mathbf{k}^b = \begin{bmatrix} p & q & r\end{bmatrix}^\top$.

Difficulty relating $\mathbf{\omega}$ with the orientation expressed with Euler angles arises because each Euler angle is defined in a different frame. Therefore, the derivatives of each Euler angle must be transformed into the proper frame as follows:

$$
\begin{align}
\mathbf{\omega} =
  \begin{bmatrix} p \\ q \\ r \end{bmatrix} &=
  \begin{bmatrix} \dot{\phi} \\ 0 \\ 0 \end{bmatrix} +
  R_{v2}^b(\phi) \begin{bmatrix} 0 \\ \dot{\theta} \\ 0 \end{bmatrix} +
  R_{v2}^b(\phi) R_{v1}^{v2}(\theta) \begin{bmatrix} 0 \\ 0 \\ \dot{\psi} \end{bmatrix} \\
&=
  \begin{bmatrix} \dot{\phi} \\ 0 \\ 0 \end{bmatrix} +
  \begin{bmatrix}
    1 & 0 & 0 \\
    0 & \cos\phi & \sin\phi \\
    0 & -\sin\phi & \cos\phi
  \end{bmatrix}
  \begin{bmatrix} 0 \\ \dot{\theta} \\ 0 \end{bmatrix} +
  \begin{bmatrix}
    1 & 0 & 0 \\
    0 & \cos\phi & \sin\phi \\
    0 & -\sin\phi & \cos\phi
  \end{bmatrix}
  \begin{bmatrix}
    \cos\theta & 0 & -\sin\theta \\
    0 & 1 & 0 \\
    \sin\theta & 0 & \cos\theta
  \end{bmatrix}
  \begin{bmatrix} 0 \\ 0 \\ \dot{\psi} \end{bmatrix} \\
&=
\underbrace{
  \begin{bmatrix}
    1 & 0 & -\sin\theta \\
    0 & \cos\phi & \sin\phi\cos\theta \\
    0 & -\sin\phi & \cos\phi\cos\theta
  \end{bmatrix}
}_{\Gamma^{-1}}
  \begin{bmatrix} \dot\phi \\ \dot\theta \\ \dot\psi \end{bmatrix}.
\end{align}
$$

Note that $\det\Gamma^{-1} = cos\theta$. Thus, for $\theta \in (-\pi/2, \pi/2)$ the matrix $\Gamma^{-1}$ is invertible, and we can write the rotational kinematic equation using Euler angles as

$$
\mathbf{\dot\Phi} =
\begin{bmatrix} \dot\phi \\ \dot\theta \\ \dot\psi \end{bmatrix} =
\begin{bmatrix}
  1 & \sin\phi\tan\theta & \cos\phi\tan\theta \\
  0 & \cos\phi & -\sin\phi \\
  0 & \sin\phi\sec\theta & \cos\phi\sec\theta
\end{bmatrix}
\begin{bmatrix} p \\ q \\ r \end{bmatrix} =
\Gamma(\phi,\theta) \mathbf{\omega}.
$$

### Dynamics

Dynamics describe the motion of objects in relation to forces and torques, which cause motion. This is often how we use a controller to interact with a physical system. This is because we can control actuators which cause a force or torque on the system.

#### Translational

It is easiest to express Newton's second law in the body frame. However, Newton's laws only hold in inertial frames. Therefore, we use the equation of Coriolis to express the derivative of an inertial frame quantity to the derivative in the body frame, which gives us

$$
\mathbf{F}_g - \mathbf{T} = m\dot{\mathbf{v}}^b + \mathbf{\omega} \times \mathbf{v}^b,
$$

where $\mathbf{F}_g = mg \begin{bmatrix} -\sin\theta & \cos\theta\sin\phi & \cos\theta\cos\phi \end{bmatrix}^\top$ and $\mathbf{T} = \begin{bmatrix} 0 & 0 & f \end{bmatrix}^\top$. This can be rearranged into

$$
\dot{\mathbf{v}}^b = \frac{1}{m}[\mathbf{F}_g - \mathbf{T} - \mathbf{\omega} \times \mathbf{v}^b].
$$

#### Translational (inertial)

Alternatively, we can express the translational dynamics in the inertial frame, which can be advantageous if we are trying to command inertial positions or velocities.

$$
m\ddot{\mathbf{r}}^i = mg\mathbf{k}^i - R_b^i \begin{bmatrix} 0 \\ 0 \\ u_1 \end{bmatrix}
$$

#### Translational (inertial) with Drag

We make the model more realistic by adding drag, see [3]. It is ideal to have a higher fidelity plant model than the control/estimation models---this will certainly be true in real life.

$$
m\ddot{\mathbf{r}}^i = mg\mathbf{k}^i -
R_b^i \begin{bmatrix} 0 \\ 0 \\ u_1 \end{bmatrix} -
\begin{bmatrix} \mu \\ \mu \\ 0 \end{bmatrix} \dot{\mathbf{r}}^i
$$

#### Rotational

Following Netwton's second law for rotational motion ($M = I\dot\omega$), we get

$$
I\mathbf{\dot\omega} = -\omega \times I\omega + M.
$$

We assume that the quadrotor is symmetric about its axes so that
$$
I=
\begin{bmatrix}
    I_x &  0  &  0  \\
     0  & I_y &  0  \\
     0  &  0  & I_z \\
\end{bmatrix}
$$

## Forces and Moments

Given that we have a dynamic model of a quadrotor, we must determine the forces and moments that are used to drive the 6 DOF dynamic model we have derived.

$$
\begin{bmatrix} u_1 \\ u_2 \\ u_3 \\ u_4 \end{bmatrix} =
\begin{bmatrix}
    1 & 1 & 1 & 1 \\
    0 & -\ell & 0 & \ell \\
    \ell & 0 & -\ell & 0 \\
    -c_T & c_T & -c_T & c_T \\
\end{bmatrix}
\begin{bmatrix} T_1 \\ T_2 \\ T_3 \\ T_4 \end{bmatrix}
$$

The thrust $T_i$ produced by each motor/propellor combination is related to the propellor's angular velocity by
$$
T_i = k\Omega_i^2.
$$

Using these expressions, we can find the actual motor commands.

Note: Given that the thrust is such that it is always overcoming gravity, a positive $m_\phi$ yields a positive motion along $\mathbf{j}^b$. A positive $m_\theta$ yields a negative motion along $\mathbf{i}^b$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

np.set_printoptions(formatter={'float': '{: 0.3f}'.format})

class Quadrotor(object):
    """Quadrotor

    This class models the physical quadrotor vehicle evolving in SE(3).
    """
    def __init__(self, r=None, Phi=None, v=None, omega=None):

        # internal quadrotor state
        self.r = r if r is not None else np.zeros((3,1))
        self.Phi = Phi if Phi is not None else np.zeros((3,1))
        self.v = v if v is not None else np.zeros((3,1))
        self.omega = omega if omega is not None else np.zeros((3,1))

        # phyiscal true parameters
        self.g = 9.81
        self.mass = 3.81
        Jxx = 0.060224; Jyy = 0.122198; Jzz = 0.132166
        self.I = np.array([[Jxx,0,0],
                           [0,Jyy,0],
                           [0,0,Jzz]])

        self.Mu = np.diag(np.array([0.85,0.85,0.85])) # drag
#         self.Mu = np.diag(np.array([0,0,0])) # drag

        # max control actuation
        self.Tmax = 40
        self.Mmax = 2

        # convenience
        self.Niters = 0

    def __str__(self):
        s  = "Quadrotor state after {} iters:\n".format(self.Niters)
        s += "\tr:     {}.T\n".format(self.r.T)
        s += "\tPhi:   {}.T\n".format(self.Phi.T)
        s += "\tv:     {}.T\n".format(self.v.T)
        s += "\tomega: {}.T\n".format(self.omega.T)
        return s

    @staticmethod
    def clamp(v, limit):
        v = np.copy(v)
        v[np.abs(v) > limit] = np.sign(v[np.abs(v) > limit])*limit
        return v

    @staticmethod
    def Gamma(phi, theta):
        return np.array([
            [1, np.sin(phi)*np.tan(theta), np.cos(phi)*np.tan(theta)],
            [0,         np.cos(phi)      ,        -np.sin(phi)      ],
            [0, np.sin(phi)/np.cos(theta), np.cos(phi)/np.cos(theta)]
        ])

    def Fg(self, phi, theta):
        return self.mass*self.g*np.array([[      -np.sin(theta)     ],
                                          [np.cos(theta)*np.sin(phi)],
                                          [np.cos(theta)*np.cos(phi)]])

    def update(self, u, dt):
        # We've been through another iteration
        self.Niters += 1

        # Extract Euler angles for convenience
        ph = self.Phi.flatten()[0]
        th = self.Phi.flatten()[1]
        ps = self.Phi.flatten()[2]

        #
        # Forces and Moments
        #

        u = np.array(u).flatten()
        T = np.array([[0, 0, u[0]]]).T
        M = np.array([[u[1], u[2], u[3]]]).T

        #
        # Saturate control effort
        #

        T = self.clamp(T, self.Tmax)
        M = self.clamp(M, self.Mmax)

        #
        # Kinematics
        #

        # Translational
#         f = lambda r: Rot_i_to_b(ph,th,ps).T.dot(self.v) # body velocities
        f = lambda r: self.v # inertial velocities
        self.r = rk4(f, self.r, dt)

        # Rotational
        f = lambda Phi: self.Gamma(Phi[0], Phi[1]).dot(self.omega)
        self.Phi = rk4(f, self.Phi, dt)

        #
        # Dynamics
        #

        # Translational
#         f = lambda v: (1/self.mass)*(self.Fg(ph,th) - T - np.cross(self.omega, v, axis=0)) # body
        f = lambda v: (1/self.mass)*(self.Fg(0,0) - Rot_i_to_b(ph,th,ps).T.dot(T) - self.Mu.dot(v)) # inertial
        self.v = rk4(f, self.v, dt)

        # Rotational
        f = lambda omega: np.linalg.inv(self.I).dot((-np.cross(omega, self.I.dot(omega), axis=0) + M))
        self.omega = rk4(f, self.omega, dt)

        # update control input
        u = np.hstack((T[2], M.flatten()))
        return u
```

```python
%%capture
%run utils.ipynb
```

```python
quad = Quadrotor()

# feed-forward thrust
z_ff = quad.mass*quad.g

Ts = 0.01
Tf = 1.5
N = int(Tf/Ts)
for i in range(N):
    m1 = 10 if Ts*i < 1 else 0
    quad.update([z_ff, 0.0, 0.0, 0.0], Ts)
print(quad)
```

    Quadrotor state after 150 iters:
    	r:     [[ 0.000  0.000  0.000]].T
    	Phi:   [[ 0.000  0.000  0.000]].T
    	v:     [[ 0.000  0.000  0.000]].T
    	omega: [[ 0.000  0.000  0.000]].T

## References

[1] R. W. Beard and T. W. McLain, Small Unmanned Aircraft. Princeton, New Jersey: Princeton University Press, 2012.

[2] R. W. Beard and T. W. Mclain, “Introduction to Feedback Control using Design Studies,” 2016.

[3] R. C. Leishman, J. C. MacDonald, R. W. Beard, and T. W. McLain, “Quadrotors and accelerometers: State estimation with an improved dynamic model,” IEEE Control Syst., vol. 34, no. 1, pp. 28–41, 2014.


# 2. Modelling with FD

The previous section studied how we can use the train-validate split with our own estimator to choose the best hyperparameter.  In this section, we want to investigate how FD can be used for ridge regression, while comparing to our previous approach.  We will use the same basic setup.  Hopefully, using FD doesn't change the regularisation too much - this will be a consequence of an accurate sketch.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.linear_model import Ridge
from sklearn.metrics import make_scorer
from sklearn.metrics import mean_squared_error
```

```python
fpath = 'datasets/'
transformed = np.load(fpath+'transformed_training_set.npy')#np.load(fpath+'train_transformed.npy')
train = np.load(fpath+'train.npy')
valid = np.load(fpath+'validate.npy')
test  = np.load(fpath+'test.npy')

X_tr   , y_tr    = transformed[:,:-1],transformed[:,-1]
X_train, y_train = train[:,:-1], train[:,-1]
X_valid, y_valid = valid[:,:-1], valid[:,-1]
X_test,  y_test   = test[:,:-1], test[:,-1]

print(f'Transformed sizes: {X_tr.shape, y_tr.shape}')
print(f'Training sizes: {X_train.shape, y_train.shape}')
print(f'Validation size: {X_valid.shape, y_valid.shape}')
print(f'Testing size: {X_test.shape, y_test.shape}')
```

    Transformed sizes: ((16512, 16), (16512,))
    Training sizes: ((13209, 16), (13209,))
    Validation size: ((3303, 16), (3303,))
    Testing size: ((4128, 16), (4128,))

### 3.1 More preparation: Polynomial Features
Now that we have some rough version of cross-validation working, we need to get the data ready for frequent directions.  Currently, the dimensionality is too low, so we will add polynomial features.  Our data currently has dimensionality $16$ - if we expand into a polynomial feature space of degree three this will become
$${16 \choose 3} = 560 $$
which is much more in the realm when FD is useful.
We can also inspect the spectra to see if this decays quickly enough.

Other feature maps are also possible (e.g. random fourier features).

```python
from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import StandardScaler
```

```python
X_train_poly = PolynomialFeatures(degree=3).fit_transform(X_train)
X_val_poly   = PolynomialFeatures(degree=3).fit_transform(X_valid)
X_test_poly  = PolynomialFeatures(degree=3).fit_transform(X_test)
```

```python
_, S_lin, _ = np.linalg.svd(X_train,full_matrices=False)
_, S_poly, _ = np.linalg.svd(X_train_poly,full_matrices=False)
```

```python
fig,ax = plt.subplots()
ax.plot(range(len(S_lin)), S_lin,label='Raw Data')
ax.plot(range(len(S_poly)), S_poly,label='Polynomial Features')
ax.legend()
ax.set_xscale('log')
ax.set_yscale('log')
```

Now let's see how the cross validation behaves:

```python
def ridge_loss(A,x,y,gamma):
    return (0.5*np.linalg.norm(A@x - y)**2 + 0.5*gamma*np.linalg.norm(x))/len(A)
```

```python
import scipy.sparse as sp
from scipy import linalg
from scipy import sparse
from scipy.sparse import linalg as sp_linalg

class RidgeRegression:
    def __init__(self,gamma):
        self.gamma = gamma

    def _solve_svd(self, X, y, alpha):
        U, s, Vt = linalg.svd(X, full_matrices=False)
        idx = s > 1e-15  # same default value as scipy.linalg.pinv
        s_nnz = s[idx][:, np.newaxis]
        UTy = np.dot(U.T, y)[:,np.newaxis]
        d = np.zeros((s.size, alpha.size), dtype=X.dtype)
        d[idx] = s_nnz / (s_nnz ** 2 + alpha)
        d_UT_y = np.squeeze(d * UTy) # squeeze to a single dimension
        sol = Vt.T@d_UT_y
        return sol#np.dot(Vt.T, d_UT_y)

    def _ridge_solve(self, X,y,alpha):
        # There should be either 1 or n_targets penalties
        alpha = np.asarray(alpha, dtype=X.dtype).ravel()
        assert alpha.size == 1
        return self._solve_svd(X,y,alpha)

    def _preprocess_data(self,X, y, fit_intercept, normalize=False):
        """Center and scale data.
        Centers data to have mean zero along axis 0.
        If fit_intercept=False or if
        the X is a sparse matrix, no centering is done, but normalization can still
        be applied. The function returns the statistics necessary to reconstruct
        the input data, which are X_offset, y_offset, X_scale, such that the output
            X = (X - X_offset) / X_scale
        X_scale is the L2 norm of X - X_offset.
        This is here because nearly all linear models will want their data to be
        centered. This function also systematically makes y consistent with X.dtype
        """
        n,d = X.shape
        self.n_targets = d
        y = np.asarray(y, dtype=X.dtype)
        if fit_intercept:
            X_offset = np.mean(X,axis=0)
            X -= X_offset
            if normalize:
                X_scale = np.linalg.norm(X,axis=0)
            else:
                X_scale = np.ones(X.shape[1], dtype=X.dtype)
            y_offset = np.mean(y,axis=0)
            y = y - y_offset
        else:
            # Return zero arrays for the offsets and ones for the scales
            X_offset = np.zeros(X.shape[1],dtype=X.dtype)
            X_scale = np.ones_like(X_offset)
            if y.ndim == 1:
                y_offset = X.dtype.type(0)
            else:
                y_offset = np.zeros(y.shape[1],dtype=X.dtype)
        return X,y, X_offset, y_offset, X_scale

    def _set_intercept(self, X_offset, y_offset, X_scale):
        '''
        Sets the intercept of the model
        '''
        self.coef_ = self.coef_ / X_scale
        self.intercept_ = y_offset - X_offset@self.coef_

    def fit(self,A,y):
        '''
        Fits the ridge model to gamma (which can be a single float or list/ndarray).
        '''
        n,d = A.shape
        # Centering the arrays for consistency with sklearn.
        X,y,X_offset, y_offset, X_scale = self._preprocess_data(A,y,fit_intercept=True)
        weights = self._ridge_solve(X,y,self.gamma)
        self.coef_ = weights
        self._set_intercept(X_offset, y_offset, X_scale)

    def predict(self,X,weights):
        return X@weights + self.intercept_

    def error(self, y_pred, y_true):
        return (1./np.sqrt(len(y_pred)))*np.linalg.norm(y_pred - y_true)

```

```python
gammas = [0] + [2**i for i in range(-15,15)]
poly_training_errors = np.zeros(len(gammas),dtype=float)
poly_validation_errors = np.zeros_like(poly_training_errors)
poly_test_errors = np.zeros_like(poly_training_errors)
for i,gi in enumerate(gammas):
    pridge = RidgeRegression(gi)
    pridge.fit(X_train_poly,y_train)
    y_train_pred = pridge.predict(X_train_poly,pridge.coef_)
    train_error = pridge.error(y_train_pred, y_train)
    y_train_val = pridge.predict(X_val_poly,pridge.coef_)
    val_error = pridge.error(y_train_val, y_valid)
    print(f'Gamma:{gi:.3E}\t\tTrainError:{train_error:.5E}\t\tValiError:{val_error:.5E}')
    poly_training_errors[i] = train_error
    poly_validation_errors[i] = val_error
#plt.plot(gammas,training_errors)
```

    Gamma:0.000E+00		TrainError:8.31051E+06		ValiError:3.14353E-02
    Gamma:3.052E-05		TrainError:1.88577E-03		ValiError:5.45246E-02
    Gamma:6.104E-05		TrainError:1.88578E-03		ValiError:5.43035E-02
    Gamma:1.221E-04		TrainError:1.88579E-03		ValiError:5.38791E-02
    Gamma:2.441E-04		TrainError:1.88585E-03		ValiError:5.30900E-02
    Gamma:4.883E-04		TrainError:1.88599E-03		ValiError:5.16933E-02
    Gamma:9.766E-04		TrainError:1.88630E-03		ValiError:4.93971E-02
    Gamma:1.953E-03		TrainError:1.88687E-03		ValiError:4.60360E-02
    Gamma:3.906E-03		TrainError:1.88785E-03		ValiError:4.19318E-02
    Gamma:7.812E-03		TrainError:1.88942E-03		ValiError:3.80133E-02
    Gamma:1.562E-02		TrainError:1.89167E-03		ValiError:3.51250E-02
    Gamma:3.125E-02		TrainError:1.89462E-03		ValiError:3.33003E-02
    Gamma:6.250E-02		TrainError:1.89826E-03		ValiError:3.19875E-02
    Gamma:1.250E-01		TrainError:1.90250E-03		ValiError:3.06335E-02
    Gamma:2.500E-01		TrainError:1.90720E-03		ValiError:2.90338E-02
    Gamma:5.000E-01		TrainError:1.91239E-03		ValiError:2.73212E-02
    Gamma:1.000E+00		TrainError:1.91827E-03		ValiError:2.56545E-02
    Gamma:2.000E+00		TrainError:1.92521E-03		ValiError:2.40220E-02
    Gamma:4.000E+00		TrainError:1.93379E-03		ValiError:2.23490E-02
    Gamma:8.000E+00		TrainError:1.94500E-03		ValiError:2.06101E-02
    Gamma:1.600E+01		TrainError:1.96027E-03		ValiError:1.88067E-02
    Gamma:3.200E+01		TrainError:1.98095E-03		ValiError:1.69114E-02
    Gamma:6.400E+01		TrainError:2.00762E-03		ValiError:1.47971E-02
    Gamma:1.280E+02		TrainError:2.04047E-03		ValiError:1.24906E-02
    Gamma:2.560E+02		TrainError:2.08072E-03		ValiError:1.05936E-02
    Gamma:5.120E+02		TrainError:2.13142E-03		ValiError:9.70694E-03
    Gamma:1.024E+03		TrainError:2.19792E-03		ValiError:9.53715E-03
    Gamma:2.048E+03		TrainError:2.28710E-03		ValiError:9.56971E-03
    Gamma:4.096E+03		TrainError:2.40168E-03		ValiError:9.65714E-03
    Gamma:8.192E+03		TrainError:2.53649E-03		ValiError:9.78981E-03
    Gamma:1.638E+04		TrainError:2.68633E-03		ValiError:9.93623E-03

```python
# Optimal regularisation:
opt_gamma = gammas[np.argmin(poly_validation_errors[1:])+1]

fig, axes = plt.subplots(nrows=2,dpi=200)
ax_t, ax_v = axes
ax_t.plot(gammas[1:],poly_training_errors[1:],label='Training')
#ax_t.plot([opt_gamma,opt_gamma],[training_errors[1:].min(),training_errors[1:].max()],label=f'OptGamma:{opt_gamma}')
ax_v.plot(gammas[1:],poly_validation_errors[1:],label='Validating')

for ax in [ax_t,ax_v]:
    if ax == ax_t:
        val_min, val_max = poly_training_errors[1:].min(),poly_training_errors[1:].max()
    else:
        val_min, val_max = poly_validation_errors[1:].min(),poly_validation_errors[1:].max()
    ax.plot([opt_gamma,opt_gamma],[val_min, val_max],label=f'OptGamma:{opt_gamma}')
    ax.set_yscale('log')
    ax.set_xscale('log',basex=2)
    ax.set_xlabel(r'$\gamma$')
    ax.legend()
```

The above plot seems to make sense: since we have expanded the feature space, we are more prone to overfitting and a way to remedy this is to regularise more.

### 3.2 Frequent Directions for Regression

Now that we have an interface that roughly corresponds with sklearn we can begin to start testing FD for model fitting.  This approach needs to roughly correspond with the class that I have already written so should have the following methods:
- Solve
- preprocess (a question here is whether we process on the raw data or on the sketch?)
- fit
- predict
- intercept
- error

The `fit` method is just going to be a wrapper around the internal workings of `preprocess, set_intercept, preprocess` so it's important to think about how all of this is going to interact.

The other thing to be aware of is that DataSketches only has implemented the [sketching technique](https://github.com/apache/incubator-datasketches-vector/blob/32c1443774472bb7382b7d88fc22d044c8808dc3/src/main/java/org/apache/datasketches/vector/decomposition/FrequentDirections.java#L348) and no further functionality for regression.

In light of this, we will import the sketches under the assumption that there is a good implementation of the sketch
algorithm itself, i.e. we can just call $B = FD(A)$ for some input $A$ to get the sketch $B$.

I will write this so it roughly follows what I already have, along with some extra methods where necessary.

```python
import sys
import os
from pathlib import Path
```

```python
path = Path(os.getcwd())
sys.path.append(str(path.parent.parent/ 'src/frequent_directions.py'))
```

    PosixPath('/Users/charlied/Google Drive/FrequentDirectionsRidgeRegression/notebooks/CaliforniaHousing')

```python
from frequent_directions import FastFrequentDirections,RobustFrequentDirections
```

```python
#RobustFrequentDirections??
```

```python
class FDRidge:

    def __init__(self, fd_dim:int,fd_mode='FD',gamma=1.0,solve_method='Exact'):
        """
        Approximate ridge regression using the FD sketch.

        fd_dim (int) - the number of rows retained in the FD sketch.
        fd_mode (str) : mode for frequent directions FD or RFD.
        alpha : float - the regularisation parameter for ridge regression.
        """
        self.fd_dim       = fd_dim
        self.fd_mode      = fd_mode
        if self.fd_mode not in ['FD', 'RFD']:
            raise NotImplementedError('Only F(ast) and R(obust) FD methods are supported.')
        self.gamma        = gamma
        self.solve_method = solve_method
        if self.solve_method not in ['Exact','Shi', 'Woodbury']:
            raise NotImplementedError('Only Exact, Shi, Woodbury methods are implemented')
        self.is_fitted    = False

    def _sketch(self,X):
        if self.fd_mode == 'FD':
            sketcher = FastFrequentDirections(X.shape[1],m=self.fd_dim)
        elif self.fd_mode == 'RFD':
            sketcher = RobustFrequentDirections(X.shape[1],m=self.fd_dim)
        sketcher.fit(X,batch_size=self.fd_dim)
        self.sketch_mat = sketcher.sketch
        self.alpha = sketcher.delta # == 0 if using FastFrequentDirections so can use self.gamma + self.alpha everywhere
        self.is_fitted = True

    def get_sketch_bytes(self):
        return self.sketch_mat.nbytes

    def get_sketch_size(self):
        return self.sketch_mat.shape

    def my_small_space_solve(self,X,y):
        '''
        Uses Woodbury to instead of evaluating $B^T B + gamma I)^{-1}
        '''
        self._sketch(X)
        B = self.sketch_mat
        BATy = B@(X.T@y)
        inv_term = np.linalg.pinv(np.eye(self.fd_dim) + B@B.T)
        right_term = inv_term@BATy
        full_inv_term = (1./self.gamma)*(X.T@y) - (B.T/self.gamma)@right_term
        return full_inv_term

    def shi_small_space_solve(self,X,y):
        '''
        Solves in small space using the algorithm of shi and phillips
        '''
        ATy = X.T@y
        self._sketch(X)
        B = self.sketch_mat
        _, S, Vt = np.linalg.svd(B,full_matrices=False)
        V = Vt.T
        inv_diag = np.linalg.pinv(np.diag(S**2 + self.gamma)) #1./(S**2 + self.gamma)
        first_term = (V@(inv_diag))@(Vt@ATy)
        second_term = (1./self.gamma)*ATy
        third_term = (1./self.gamma)*V@(Vt@ATy)
        return first_term + second_term - third_term

    def _solve(self, X, y):
        '''
        Obtains the inverse term explicitly
        '''
        if self.solve_method == 'Shi':
            return self.shi_small_space_solve(X,y)
        elif self.solve_method == 'Woodbury':
            return self.my_small_space_solve(X,y)
        else:
            self._sketch(X)
            return (np.linalg.pinv(self.sketch_mat.T@self.sketch_mat + self.gamma*np.eye(X.shape[1])))@(X.T@y)

    def _preprocess_data(self,X, y, fit_intercept, normalize=False):
        """Center and scale data.
        Centers data to have mean zero along axis 0.
        If fit_intercept=False or if
        the X is a sparse matrix, no centering is done, but normalization can still
        be applied. The function returns the statistics necessary to reconstruct
        the input data, which are X_offset, y_offset, X_scale, such that the output
            X = (X - X_offset) / X_scale
        X_scale is the L2 norm of X - X_offset.
        This is here because nearly all linear models will want their data to be
        centered. This function also systematically makes y consistent with X.dtype
        """
        n,d = X.shape
        self.n_targets = d
        y = np.asarray(y, dtype=X.dtype)
        if fit_intercept:
            X_offset = np.mean(X,axis=0)
            X -= X_offset
            if normalize:
                X_scale = np.linalg.norm(X,axis=0)
            else:
                X_scale = np.ones(X.shape[1], dtype=X.dtype)
            y_offset = np.mean(y,axis=0)
            y = y - y_offset
        else:
            # Return zero arrays for the offsets and ones for the scales
            X_offset = np.zeros(X.shape[1],dtype=X.dtype)
            X_scale = np.ones_like(X_offset)
            if y.ndim == 1:
                y_offset = X.dtype.type(0)
            else:
                y_offset = np.zeros(y.shape[1],dtype=X.dtype)
        return X,y, X_offset, y_offset, X_scale

    def _set_intercept(self, X_offset, y_offset, X_scale):
        '''
        Sets the intercept of the model
        '''
        self.coef_ = self.coef_ / X_scale
        self.intercept_ = y_offset - X_offset@self.coef_

    def fit(self,A,y):
        '''
        Fits the ridge model to gamma (which can be a single float or list/ndarray).
        '''
        n,d = A.shape
        # Centering the arrays for consistency with sklearn.
        X,y,X_offset, y_offset, X_scale = self._preprocess_data(A,y,fit_intercept=True)
        weights = self._solve(X,y)
        self.coef_ = weights
        self._set_intercept(X_offset, y_offset, X_scale)

    def predict(self,X,weights):
        return X@weights + self.intercept_

    def error(self, y_pred, y_true):
        return (1./np.sqrt(len(y_pred)))*np.linalg.norm(y_pred - y_true)
```

First, let's establish that there is reasonable accuracy from the regression task using FD.

```python
fd_ridge = FDRidge(fd_dim=256,fd_mode='RFD',gamma=256.)
fd_ridge.fit(X_train_poly,y_train)

ridge = RidgeRegression(256.)
ridge.fit(X_train_poly,y_train)
```

```python
fd_error = np.linalg.norm(ridge.coef_ - fd_ridge.coef_)
print(f'FD Regression sketching error: {fd_error:.6f}')
```

    FD Regression sketching error: 0.001990

```python
fig, ax = plt.subplots()
ax.scatter(range(len(fd_ridge.coef_)),fd_ridge.coef_,label='FD')
ax.scatter(range(len(ridge.coef_)),ridge.coef_,label='Exact')
ax.set_ylim(-0.0025,0.0025)
```

One thing to consider here is that because of the shrinkage applied to the spectrum, the directions in FDRR need boosting slightly.  This is achieved by using a slightly larger regularisation parameter $\gamma$.

### 3.3 Different methods for solving

There are different solutions that can be returned.  We have the standard solution:
\begin{align}
\text{[Standard]} \quad x_S &= (B^T B + \gamma I)^{-1} X^T y \\
\text{[Woodbury]} \quad x_W &= \frac1\gamma \left[I_d - \frac{B^T}{\gamma}(I_m + BB^T)^{-1}B \right] X^T y \\
\text{[EigenVec]} \quad x_E &= \left[V(\Sigma^2 +\gamma I_m)^{-1}V^T + \frac1\gamma (I_d - VV^T)  \right]X^T y \\
\end{align}

Eigenvec and Woodbury are the same up to determining whether to use the basis or the sketch itself.  This can be shown by using the SVD of $B = U \Sigma V^T$ and decomposing the entire product to show equivalence.  Let's plot the performance of the FD methods vs the exact method.

```python
# Solve using FD - Exact solver

gammas = [0] + [2**i for i in range(-15,15)]
fd_poly_training_errors = np.zeros(len(gammas),dtype=float)
sketch_nbytes = np.zeros_like(fd_poly_training_errors)
fd_poly_validation_errors = np.zeros_like(fd_poly_training_errors)
fd_poly_test_errors = np.zeros_like(fd_poly_training_errors)
for i,gi in enumerate(gammas):
    fd_ridge = FDRidge(fd_dim=256,fd_mode='RFD',gamma=gi)
    fd_ridge.fit(X_train_poly,y_train)
    y_train_pred = fd_ridge.predict(X_train_poly,fd_ridge.coef_)
    train_error = fd_ridge.error(y_train_pred, y_train)
    y_train_val = fd_ridge.predict(X_val_poly,fd_ridge.coef_)
    val_error = fd_ridge.error(y_train_val, y_valid)
    print(f'Gamma:{gi:.3E}\t\tTrainError:{train_error:.5E}\t\tValiError:{val_error:.5E}')
    fd_poly_training_errors[i] = train_error
    fd_poly_validation_errors[i] = val_error
    sketch_nbytes[i] = fd_ridge.get_sketch_bytes()
```

    Gamma:0.000E+00		TrainError:5.60652E-02		ValiError:6.73956E-01
    Gamma:3.052E-05		TrainError:5.60551E-02		ValiError:6.73872E-01
    Gamma:6.104E-05		TrainError:5.60456E-02		ValiError:6.73785E-01
    Gamma:1.221E-04		TrainError:5.60351E-02		ValiError:6.73690E-01
    Gamma:2.441E-04		TrainError:5.60081E-02		ValiError:6.73452E-01
    Gamma:4.883E-04		TrainError:5.59548E-02		ValiError:6.72969E-01
    Gamma:9.766E-04		TrainError:5.58511E-02		ValiError:6.72059E-01
    Gamma:1.953E-03		TrainError:3.77902E+01		ValiError:4.23283E+02
    Gamma:3.906E-03		TrainError:1.89184E+01		ValiError:2.11615E+02
    Gamma:7.812E-03		TrainError:9.45353E+00		ValiError:1.06000E+02
    Gamma:1.562E-02		TrainError:4.72513E+00		ValiError:5.31422E+01
    Gamma:3.125E-02		TrainError:2.36393E+00		ValiError:2.67086E+01
    Gamma:6.250E-02		TrainError:1.18331E+00		ValiError:1.34871E+01
    Gamma:1.250E-01		TrainError:5.93026E-01		ValiError:6.86870E+00
    Gamma:2.500E-01		TrainError:2.97699E-01		ValiError:3.54705E+00
    Gamma:5.000E-01		TrainError:1.49662E-01		ValiError:1.87042E+00
    Gamma:1.000E+00		TrainError:7.52957E-02		ValiError:1.01431E+00
    Gamma:2.000E+00		TrainError:3.79079E-02		ValiError:5.68880E-01
    Gamma:4.000E+00		TrainError:1.91417E-02		ValiError:3.31576E-01
    Gamma:8.000E+00		TrainError:9.78615E-03		ValiError:2.01796E-01
    Gamma:1.600E+01		TrainError:5.21834E-03		ValiError:1.27293E-01
    Gamma:3.200E+01		TrainError:3.11654E-03		ValiError:8.02487E-02
    Gamma:6.400E+01		TrainError:2.28737E-03		ValiError:4.77969E-02
    Gamma:1.280E+02		TrainError:2.05424E-03		ValiError:2.59790E-02
    Gamma:2.560E+02		TrainError:2.04203E-03		ValiError:1.40346E-02
    Gamma:5.120E+02		TrainError:2.09480E-03		ValiError:1.00933E-02
    Gamma:1.024E+03		TrainError:2.17056E-03		ValiError:9.55315E-03
    Gamma:2.048E+03		TrainError:2.26817E-03		ValiError:9.56467E-03
    Gamma:4.096E+03		TrainError:2.38990E-03		ValiError:9.64881E-03
    Gamma:8.192E+03		TrainError:2.52984E-03		ValiError:9.78346E-03
    Gamma:1.638E+04		TrainError:2.68266E-03		ValiError:9.93313E-03

```python
print(f'Input array size: {X_train.shape} bytes: {X_train.nbytes}')
print(f'Sketch rows: 256; bytes: {sketch_nbytes}')
```

    Input array size: (13209, 16) bytes: 1690752
    Sketch rows: 256; bytes: [1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512.
     1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512.
     1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512.
     1984512. 1984512. 1984512. 1984512. 1984512. 1984512. 1984512.]

```python
fd_ridge = FDRidge(fd_dim=256,fd_mode='RFD',gamma=gi)
fd_ridge.fit(X_train_poly,y_train)
print(fd_ridge.get_sketch_bytes(), fd_ridge.get_sketch_size())
```

    1984512 (256, 969)

```python
print('Using FD results in a {:.1f}x factor space saving in rows and bytes'.format(X_train_poly.shape[0] / fd_ridge.get_sketch_size()[0] / fd_ridge.get_sketch_size()[0]))
```

    Using FD results in a 51.6x factor space saving in rows and bytes

This is equivalent to saying that the summary is 2% the size of the input data, and provided that the hyperparameter is appropriately set, yields equivalent validation performance.

```python
print(f'Input rows:{X_train_poly.shape[0]}\t0.02 x input rows:{fd_ridge.get_sketch_size()[0]}')
```

    Input rows:13209	0.02 x input rows:256

```python
# Optimal regularisation:
opt_gamma = gammas[np.argmin(poly_validation_errors[1:])+1]
opt_fd_gamma = gammas[np.argmin(fd_poly_validation_errors[1:])+1]

fig, axes = plt.subplots(nrows=2,dpi=200)
ax_t, ax_v = axes
ax_t.plot(gammas[1:],poly_training_errors[1:],label='Exact')
ax_t.plot(gammas[1:],fd_poly_training_errors[1:],label='FDRR')
ax_t.set_ylabel('Training Error')

#ax_t.plot([opt_gamma,opt_gamma],[training_errors[1:].min(),training_errors[1:].max()],label=f'OptGamma:{opt_gamma}')
ax_v.plot(gammas[1:],poly_validation_errors[1:],label='Exact')
ax_v.plot(gammas[1:],fd_poly_validation_errors[1:],label='FDRR')
ax_v.set_ylabel('Validation Error')

for ax in [ax_t,ax_v]:
    if ax == ax_t:
        val_min, val_max = poly_training_errors[1:].min(),fd_poly_training_errors[1:].max()
    else:
        val_min, val_max = poly_validation_errors[1:].min(),fd_poly_validation_errors[1:].max()
    ax.plot([opt_gamma,opt_gamma],[val_min, val_max],label=f'OptGamma:{opt_gamma}')
    ax.set_yscale('log')
    ax.set_yscale('log',basey=10)
    ax.set_xscale('log',basex=10)
    ax.set_xlabel(r'$\gamma$')
    ax.legend(loc='upper left')
    ax.set_ylim(top=2**(-1))
fig.suptitle('Exact solver')
print(f'Optimal gamma: {opt_gamma}')
print(f'FD Optimal gamma: {opt_fd_gamma}')

```

This is only the behaviour for the exact solver, i.e. $x_S.  We can repeat the same process for Woodbury and Eigenvector $ x_W, x_E$.

```python
# Solve using FD - Using Woodbury solver

gammas = [2**i for i in range(-15,15)]
fd_poly_training_errors = np.zeros(len(gammas),dtype=float)
sketch_nbytes = np.zeros_like(fd_poly_training_errors)
fd_poly_validation_errors = np.zeros_like(fd_poly_training_errors)
fd_poly_test_errors = np.zeros_like(fd_poly_training_errors)
for i,gi in enumerate(gammas):
    fd_ridge = FDRidge(fd_dim=256,fd_mode='RFD',gamma=gi,solve_method='Woodbury')
    fd_ridge.fit(X_train_poly,y_train)
    y_train_pred = fd_ridge.predict(X_train_poly,fd_ridge.coef_)
    train_error = fd_ridge.error(y_train_pred, y_train)
    y_train_val = fd_ridge.predict(X_val_poly,fd_ridge.coef_)
    val_error = fd_ridge.error(y_train_val, y_valid)
    print(f'Gamma:{gi:.3E}\t\tTrainError:{train_error:.5E}\t\tValiError:{val_error:.5E}')
    fd_poly_training_errors[i] = train_error
    fd_poly_validation_errors[i] = val_error
    sketch_nbytes[i] = fd_ridge.get_sketch_bytes()
```

    Gamma:3.052E-05		TrainError:2.49143E+03		ValiError:3.32460E+04
    Gamma:6.104E-05		TrainError:1.24571E+03		ValiError:1.66230E+04
    Gamma:1.221E-04		TrainError:6.22856E+02		ValiError:8.31150E+03
    Gamma:2.441E-04		TrainError:3.11428E+02		ValiError:4.15575E+03
    Gamma:4.883E-04		TrainError:1.55713E+02		ValiError:2.07787E+03
    Gamma:9.766E-04		TrainError:7.78563E+01		ValiError:1.03894E+03
    Gamma:1.953E-03		TrainError:3.89277E+01		ValiError:5.19468E+02
    Gamma:3.906E-03		TrainError:1.94634E+01		ValiError:2.59734E+02
    Gamma:7.812E-03		TrainError:9.73129E+00		ValiError:1.29867E+02
    Gamma:1.562E-02		TrainError:4.86522E+00		ValiError:6.49332E+01
    Gamma:3.125E-02		TrainError:2.43219E+00		ValiError:3.24665E+01
    Gamma:6.250E-02		TrainError:1.21567E+00		ValiError:1.62331E+01
    Gamma:1.250E-01		TrainError:6.07421E-01		ValiError:8.11637E+00
    Gamma:2.500E-01		TrainError:3.03306E-01		ValiError:4.05803E+00
    Gamma:5.000E-01		TrainError:1.51270E-01		ValiError:2.02887E+00
    Gamma:1.000E+00		TrainError:7.52957E-02		ValiError:1.01432E+00
    Gamma:2.000E+00		TrainError:3.73966E-02		ValiError:5.07086E-01
    Gamma:4.000E+00		TrainError:1.86269E-02		ValiError:2.53567E-01
    Gamma:8.000E+00		TrainError:9.60044E-03		ValiError:1.26999E-01
    Gamma:1.600E+01		TrainError:5.70243E-03		ValiError:6.40950E-02
    Gamma:3.200E+01		TrainError:4.43041E-03		ValiError:3.33716E-02
    Gamma:6.400E+01		TrainError:4.17349E-03		ValiError:1.92446E-02
    Gamma:1.280E+02		TrainError:4.16788E-03		ValiError:1.36815E-02
    Gamma:2.560E+02		TrainError:4.19673E-03		ValiError:1.19466E-02
    Gamma:5.120E+02		TrainError:4.21892E-03		ValiError:1.15009E-02
    Gamma:1.024E+03		TrainError:4.23193E-03		ValiError:1.14013E-02
    Gamma:2.048E+03		TrainError:4.23891E-03		ValiError:1.13836E-02
    Gamma:4.096E+03		TrainError:4.24251E-03		ValiError:1.13828E-02
    Gamma:8.192E+03		TrainError:4.24435E-03		ValiError:1.13844E-02
    Gamma:1.638E+04		TrainError:4.24527E-03		ValiError:1.13858E-02

```python
# Optimal regularisation:
opt_gamma = gammas[np.argmin(poly_validation_errors[1:])+1]
opt_fd_gamma = gammas[np.argmin(fd_poly_validation_errors[1:])+1]

fig, axes = plt.subplots(nrows=2,dpi=200)
ax_t, ax_v = axes
ax_t.plot(gammas[:],poly_training_errors[1:],label='Exact')
ax_t.plot(gammas[:],fd_poly_training_errors[:],label='FDRR')
ax_t.set_ylabel('Training Error')

#ax_t.plot([opt_gamma,opt_gamma],[training_errors[1:].min(),training_errors[1:].max()],label=f'OptGamma:{opt_gamma}')
ax_v.plot(gammas[:],poly_validation_errors[1:],label='Exact')
ax_v.plot(gammas[:],fd_poly_validation_errors[:],label='FDRR')
ax_v.set_ylabel('Validation Error')

for ax in [ax_t,ax_v]:
    if ax == ax_t:
        val_min, val_max = poly_training_errors[1:].min(),fd_poly_training_errors[1:].max()
    else:
        val_min, val_max = poly_validation_errors[1:].min(),fd_poly_validation_errors[1:].max()
    ax.plot([opt_gamma,opt_gamma],[val_min, val_max],label=f'OptGamma:{opt_gamma}')
    ax.set_yscale('log')
    ax.set_yscale('log',basey=10)
    ax.set_xscale('log',basex=10)
    ax.set_xlabel(r'$\gamma$')
    ax.legend(loc='upper left')
    ax.set_ylim(top=2**(-1))
fig.suptitle('Woodbury solver')
print(f'Optimal gamma: {opt_gamma}')
print(f'FD Optimal gamma: {opt_fd_gamma}')

```

```python
# Solve using FD - Using Shi Phillips

gammas = [2**i for i in range(-15,15)]
fd_poly_training_errors = np.zeros(len(gammas),dtype=float)
sketch_nbytes = np.zeros_like(fd_poly_training_errors)
fd_poly_validation_errors = np.zeros_like(fd_poly_training_errors)
fd_poly_test_errors = np.zeros_like(fd_poly_training_errors)
for i,gi in enumerate(gammas):
    fd_ridge = FDRidge(fd_dim=256,fd_mode='RFD',gamma=gi,solve_method='Shi')
    fd_ridge.fit(X_train_poly,y_train)
    y_train_pred = fd_ridge.predict(X_train_poly,fd_ridge.coef_)
    train_error = fd_ridge.error(y_train_pred, y_train)
    y_train_val = fd_ridge.predict(X_val_poly,fd_ridge.coef_)
    val_error = fd_ridge.error(y_train_val, y_valid)
    print(f'Gamma:{gi:.3E}\t\tTrainError:{train_error:.5E}\t\tValiError:{val_error:.5E}')
    fd_poly_training_errors[i] = train_error
    fd_poly_validation_errors[i] = val_error
    sketch_nbytes[i] = fd_ridge.get_sketch_bytes()
```

    Gamma:3.052E-05		TrainError:2.41808E+03		ValiError:2.70677E+04
    Gamma:6.104E-05		TrainError:1.20904E+03		ValiError:1.35340E+04
    Gamma:1.221E-04		TrainError:6.04522E+02		ValiError:6.76714E+03
    Gamma:2.441E-04		TrainError:3.02262E+02		ValiError:3.38371E+03
    Gamma:4.883E-04		TrainError:1.51132E+02		ValiError:1.69200E+03
    Gamma:9.766E-04		TrainError:7.55671E+01		ValiError:8.46139E+02
    Gamma:1.953E-03		TrainError:3.77846E+01		ValiError:4.23210E+02
    Gamma:3.906E-03		TrainError:1.88934E+01		ValiError:2.11745E+02
    Gamma:7.812E-03		TrainError:9.44784E+00		ValiError:1.06012E+02
    Gamma:1.562E-02		TrainError:4.72511E+00		ValiError:5.31450E+01
    Gamma:3.125E-02		TrainError:2.36384E+00		ValiError:2.67091E+01
    Gamma:6.250E-02		TrainError:1.18330E+00		ValiError:1.34870E+01
    Gamma:1.250E-01		TrainError:5.93029E-01		ValiError:6.86839E+00
    Gamma:2.500E-01		TrainError:2.97697E-01		ValiError:3.54702E+00
    Gamma:5.000E-01		TrainError:1.49661E-01		ValiError:1.87042E+00
    Gamma:1.000E+00		TrainError:7.52957E-02		ValiError:1.01432E+00
    Gamma:2.000E+00		TrainError:3.79080E-02		ValiError:5.68879E-01
    Gamma:4.000E+00		TrainError:1.91417E-02		ValiError:3.31576E-01
    Gamma:8.000E+00		TrainError:9.78615E-03		ValiError:2.01796E-01
    Gamma:1.600E+01		TrainError:5.21834E-03		ValiError:1.27293E-01
    Gamma:3.200E+01		TrainError:3.11654E-03		ValiError:8.02487E-02
    Gamma:6.400E+01		TrainError:2.28737E-03		ValiError:4.77969E-02
    Gamma:1.280E+02		TrainError:2.05424E-03		ValiError:2.59790E-02
    Gamma:2.560E+02		TrainError:2.04203E-03		ValiError:1.40346E-02
    Gamma:5.120E+02		TrainError:2.09480E-03		ValiError:1.00933E-02
    Gamma:1.024E+03		TrainError:2.17056E-03		ValiError:9.55315E-03
    Gamma:2.048E+03		TrainError:2.26817E-03		ValiError:9.56467E-03
    Gamma:4.096E+03		TrainError:2.38990E-03		ValiError:9.64881E-03
    Gamma:8.192E+03		TrainError:2.52984E-03		ValiError:9.78346E-03
    Gamma:1.638E+04		TrainError:2.68266E-03		ValiError:9.93313E-03

```python
# Optimal regularisation:
opt_gamma = gammas[np.argmin(poly_validation_errors[1:])+1]
opt_fd_gamma = gammas[np.argmin(fd_poly_validation_errors[1:])+1]

fig, axes = plt.subplots(nrows=2,dpi=200)
ax_t, ax_v = axes
ax_t.plot(gammas[:],poly_training_errors[1:],label='Exact')
ax_t.plot(gammas[:],fd_poly_training_errors[:],label='FDRR')
ax_t.set_ylabel('Training Error')

#ax_t.plot([opt_gamma,opt_gamma],[training_errors[1:].min(),training_errors[1:].max()],label=f'OptGamma:{opt_gamma}')
ax_v.plot(gammas[:],poly_validation_errors[1:],label='Exact')
ax_v.plot(gammas[:],fd_poly_validation_errors[:],label='FDRR')
ax_v.set_ylabel('Validation Error')

for ax in [ax_t,ax_v]:
    if ax == ax_t:
        val_min, val_max = poly_training_errors[1:].min(),fd_poly_training_errors[1:].max()
    else:
        val_min, val_max = poly_validation_errors[1:].min(),fd_poly_validation_errors[1:].max()
    ax.plot([opt_gamma,opt_gamma],[val_min, val_max],label=f'OptGamma:{opt_gamma}')
    ax.set_yscale('log')
    ax.set_yscale('log',basey=10)
    ax.set_xscale('log',basex=10)
    ax.set_xlabel(r'$\gamma$')
    ax.legend(loc='upper left')
    ax.set_ylim(top=2**(-1))
fig.suptitle('Shi-Phillips Eigenvector solver')
print(f'Optimal gamma: {opt_gamma}')
print(f'FD Optimal gamma: {opt_fd_gamma}')
```

**Summary**:
The three methods all perform similarly.  However, the main point is that it looks like the Shi-Phillips approach is slightly more numerically stable than using Woodbury explicitly.  Probably, if more care is taken with the Woodbury implementation this can be improved.
As expected, they are both roughly similar to the exact method (which is not scalable for large $d$).

### 3.4 Understanding how regularization helps
Performance is better when $\gamma$ is increased, but why is this the case?  Hopefully the plot below should give some information as to why this is the case.  The figure shows that adding the regularization boosts all of the singular values, but the relative difference between the largest and the smallest singular values is reduced as $\gamma$ is increased.

Numerically, this make the problem more stable as the condition number is smaller.  Algorithmically, increasing $\gamma$ also makes for an easier problem for FD to solve.  This is due to the reciprocal dependence on $m = O(1/\gamma)$. Hence, the accuracy guarantee for a fixed projection dimension $m$ is better as $\gamma$ is increased.

```python
fig,ax = plt.subplots(dpi=200)

ax.plot(range(len(S_poly)), S_poly,label='Polynomial Features')
for i,gi in enumerate(gammas):
    if i%4 == 0:
        ax.plot(range(len(S_poly)), S_poly+gi,label=f'{gi:.3f}')
ax.legend(loc='lower left',ncol=2)
ax.set_xlabel(r'$\sigma_i^2 + gamma')
ax.set_xscale('log')
ax.set_yscale('log')
```

This plot suggests that $\gamma$ needs to be somewhat substantial to improve the conditioning.

```python
fig,ax = plt.subplots(dpi=200)
conds = np.zeros_like(gammas)
smin,smax = S_poly.min(), S_poly.max()
for i,gi in enumerate(gammas):
    conds[i] = (smax + gi)/(smin + gi)

#ax.plot(range(len(S_lin)), S_lin,label='Raw }Data')
ax.plot(gammas, conds)
ax.plot([opt_gamma,opt_gamma],[0., conds.max()],label=f'OptGamma:{opt_gamma}')
ax.set_ylabel('Condition number')
ax.set_xlabel(r'$\gamma$')
# for i,gi in enumerate(gammas):
#         ax.plot(range(len(S_poly)), S_poly+gi,label=f'{gi:.3f}')
ax.legend(loc='upper left',ncol=2)
ax.set_xscale('log')
ax.set_yscale('log')
```

### 3.5 Different methods for solving: Quick Comparion
Recall that we have 3 different definitions for obtaining our solution $\hat{x}$.
The last two $ x_W, x_E$ are algebraically equivalent which is easy to show from Woodbury and an SVD.  The first one though, is the exact/brute force solve on the normal equations.  Since $m < d$ this is not going to be scalable for high dimensional problems.

We can briefly elicit this behaviour by performing the following:

```python
%timeit np.linalg.pinv(X_train_poly.T@X_train_poly + 1024.*np.eye(X_train_poly.shape[1]))
```

    442 ms ± 83 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
sketcher = RobustFrequentDirections(X_train_poly.shape[1],m=256)
sketcher.fit(X_train_poly,256)
```

```python
B = sketcher.sketch
alpha = sketcher.delta
m,d = B.shape
g = 1024.
BATy = B@(X_train_poly.T@y_train)
```

```python
def woodbury_inv():
    inv_term = np.linalg.pinv(np.eye(m) + B@B.T)
    right_term = inv_term@BATy
    full_inv_term = (1./g)*(X_train_poly.T@y_train) - (B.T/g)@right_term
```

```python
%timeit woodbury_inv()
```

    14.8 ms ± 195 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

It's clear to see that even on these small test instances, Woodbury is much more scalable and time efficient.


# Custom finite difference coefficients in Devito

## Introduction

When taking the numerical derivative of a function in Devito, the default behaviour is for 'standard' finite difference weights (obtained via a Taylor series expansion about the point of differentiation) to be applied. Consider the following example for some field $u(\mathbf{x},t)$, where $\mathbf{x}=(x,y)$. Let us define a computational domain/grid and differentiate our field with respect to $x$.

```python
import numpy as np
import sympy as sp
from devito import Grid, TimeFunction

# Create our grid (computational domain)
Lx = 10
Ly = Lx
Nx = 11
Ny = Nx
dx = Lx/(Nx-1)
dy = dx
grid = Grid(shape=(Nx,Ny), extent=(Lx,Ly))

# Define u(x,y,t) on this grid
u = TimeFunction(name='u', grid=grid, time_order=2, space_order=2)

# Define symbol for laplacian replacement
H = sp.symbols('H')
```

Now, lets look at the output of $\partial u/\partial x$:

```python
print(u.dx.evaluate)
```

    -u(t, x, y)/h_x + u(t, x + h_x, y)/h_x

By default the 'standard' Taylor series expansion result, where `h_x` represents the $x$-direction grid spacing, is returned. However, there may be instances when a user wishes to use 'non-standard' weights when, for example, implementing a dispersion-relation-preserving (DRP) scheme. See e.g.

[1] Christopher K.W. Tam, Jay C. Webb (1993). ”Dispersion-Relation-Preserving Finite Difference Schemes for Computational Acoustics.” **J. Comput. Phys.**, 107(2), 262--281. https://doi.org/10.1006/jcph.1993.1142

for further details. The use of such modified weights is facilitated in Devito via the 'symbolic' finite difference coefficents functionality. Let us start by re-defining the function $u(\mathbf{x},t)$ in the following manner:

```python
u = TimeFunction(name='u', grid=grid, time_order=2, space_order=2, coefficients='symbolic')
```

Note the addition of the `coefficients='symbolic'` keyword. Now, when printing $\partial u/\partial x$ we obtain:

```python
print(u.dx.evaluate)
```

    W(x, 1, u(t, x, y), x)*u(t, x, y) + W(x - h_x, 1, u(t, x, y), x)*u(t, x - h_x, y) + W(x + h_x, 1, u(t, x, y), x)*u(t, x + h_x, y)

Owing to the addition of the `coefficients='symbolic'` keyword the weights have been replaced by sympy functions. Now, take for example the weight `W(x - h_x, 1, u(t, x, y), x)`, the notation is as follows:
* The first `x - h_x` refers to the spatial location of the weight w.r.t. the evaluation point `x`.
* The `1` refers to the order of the derivative.
* `u(t, x, y)` refers to the function with which the weight is associated.
* Finally, the `x` refers to the dimension along which the derivative is being taken.

Symbolic coefficients can then be manipulated using the Devito 'Coefficient' and 'Substitutions' objects. First, let us consider an example where we wish to replace the coefficients with a set of constants throughout the entire computational domain.

```python
from devito import Coefficient, Substitutions # Import the Devito Coefficient and Substitutions objects
# Grab the grid spatial dimensions: Note x[0] will correspond to the x-direction and x[1] to y-direction
x = grid.dimensions
# Form a Coefficient object and then a replacement rules object (to pass to a Devito equation):
u_x_coeffs = Coefficient(1, u, x[0], np.array([-0.6, 0.1, 0.6]))
coeffs = Substitutions(u_x_coeffs)
```

Devito Coefficient ojects take arguments in the following order:
1. Derivative order (in the above example this is the first derivative)
2. Function to which the coefficients 'belong' (in the above example this is the time function `u`)
3. Dimension on which coefficients will be applied (in the above example this is the x-direction)
4. Coefficient data. Since, in the above example, the coefficients have been applied as a 1-d numpy array replacement will occur at the equation level. (Note that other options are in development and will be the subject of future notebooks).

Now, lets form a Devito equation, pass it the Substitutions object, and take a look at the output:

```python
from devito import Eq
eq = Eq(u.dt+u.dx, coefficients=coeffs)
print(eq.evaluate)
```

    Eq(0.1*u(t, x, y) - 0.6*u(t, x - h_x, y) + 0.6*u(t, x + h_x, y) - u(t - dt, x, y)/(2*dt) + u(t + dt, x, y)/(2*dt), 0)

We see that in the above equation the standard weights for the first derivative of `u` in the $x$-direction have now been replaced with our user defined weights. Note that since no replacement rules were defined for the time derivative (`u.dt`) standard weights have replaced the symbolic weights.

Now, let us consider a more complete example.

## Example: Finite difference modeling for a large velocity-contrast acousitc wave model

It is advised to read through the 'Introduction to seismic modelling' notebook located in  devito/examples/seismic/tutorials/01_modelling.ipynb before proceeding with this example since much introductory material will be ommited here. The example now considered is based on an example introduced in

[2] Yang Liu (2013). ”Globally optimal finite-difference schemes based on least squares.” **GEOPHYSICS**, 78(4), 113--132. https://doi.org/10.1190/geo2012-0480.1.

See figure 18 of [2] for further details. Note that here we will simply use Devito to 'reproduce' the simulations leading to two results presented in the aforementioned figure. No analysis of the results will be carried out. The domain under consideration has a sptaial extent of $2km \times 2km$ and, letting $x$ be the horizontal coordinate and $z$ the depth, a velocity profile such that $v_1(x,z)=1500ms^{-1}$ for $z\leq1200m$ and $v_2(x,z)=4000ms^{-1}$ for $z>1200m$.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Model, plot_velocity
%matplotlib inline

# Define a physical size
Lx = 2000
Lz = Lx
h = 10
Nx = int(Lx/h)+1
Nz = Nx

shape = (Nx, Nz)  # Number of grid point
spacing = (h, h)  # Grid spacing in m. The domain size is now 2km by 2km
origin = (0., 0.)

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:, :121] = 1.5
v[:, 121:] = 4.0

# With the velocity and model size defined, we can create the seismic model that
# encapsulates these properties. We also define the size of the absorbing layer as 10 grid points
nbl = 10
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=20, nbl=nbl, bcs="damp")

plot_velocity(model)
```

The seismic wave source term will be modelled as a Ricker Wavelet with a peak-frequency of $25$Hz located at $(1000m,800m)$. Before applying the DRP scheme, we begin by generating a 'reference' solution using a spatially high-order standard finite difference scheme and time step well below the model's critical time-step. The scheme will be 2nd order in time.

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 500.  # Simulation lasts 0.5 seconds (500 ms)
dt = 1.0  # Time step of 0.2ms

time_range = TimeAxis(start=t0, stop=tn, step=dt)

#NBVAL_IGNORE_OUTPUT
from examples.seismic import RickerSource

f0 = 0.015  # Source peak frequency is 25Hz (0.025 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

# First, position source centrally in all dimensions, then set depth
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 800.  # Depth is 800m

# We can plot the time signature to see the wavelet
src.show()
```

Now let us define our wavefield and PDE:

```python
# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=20)

# We can now write the PDE
pde = model.m * u.dt2 - H + model.damp * u.dt

# This discrete PDE can be solved in a time-marching way updating u(t+dt) from the previous time step
# Devito as a shortcut for u(t+dt) which is u.forward. We can then rewrite the PDE as
# a time marching updating equation known as a stencil using customized SymPy functions
from devito import solve

stencil = Eq(u.forward, solve(pde, u.forward).subs({H: u.laplace}))
```

```python
# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)
```

Now, lets create the operator and execute the time marching scheme:

```python
from devito import Operator

op = Operator([stencil] + src_term, subs=model.spacing_map)
```

```python
#NBVAL_IGNORE_OUTPUT
op(time=time_range.num-1, dt=dt)
```

    Operator `Kernel` run in 0.06 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=0.05034199999999987, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section1', rank=None),
                         PerfEntry(time=0.0006920000000000073, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

And plot the result:

```python
#import matplotlib
import matplotlib.pyplot as plt
from matplotlib import cm

Lx = 2000
Lz = 2000

abs_lay = nbl*h

dx = h
dz = dx
X, Z = np.mgrid[-abs_lay: Lx+abs_lay+1e-10: dx, -abs_lay: Lz+abs_lay+1e-10: dz]

levels = 100

fig = plt.figure(figsize=(14, 7))
ax1 = fig.add_subplot(111)
cont = ax1.contourf(X,Z,u.data[0,:,:], levels, cmap=cm.binary)
fig.colorbar(cont)
ax1.axis([0, Lx, 0, Lz])
ax1.set_xlabel('$x$')
ax1.set_ylabel('$z$')
ax1.set_title('$u(x,z,500)$')

plt.gca().invert_yaxis()
plt.show()
```

We will now reimplement the above model applying the DRP scheme presented in [2].

First, since we wish to apply different custom FD coefficients in the upper on lower layers we need to define these two 'subdomains' using the `Devito SubDomain` functionality:

```python
from devito import SubDomain

# Define our 'upper' and 'lower' SubDomains:
class Upper(SubDomain):
    name = 'upper'
    def define(self, dimensions):
        x, z = dimensions
        # We want our upper layer to span the entire x-dimension and all
        # but the bottom 80 (+boundary layer) cells in the z-direction, which is achieved via
        # the following notation:
        return {x: x, z: ('left', 80+nbl)}

class Lower(SubDomain):
    name = 'lower'
    def define(self, dimensions):
        x, z = dimensions
        # We want our lower layer to span the entire x-dimension and all
        # but the top 121 (+boundary layer) cells in the z-direction.
        return {x: x, z: ('right', 121+nbl)}

# Create these subdomains:
ur = Upper()
lr = Lower()
```

We now create our model incoporating these subdomains:

```python
#NBVAL_IGNORE_OUTPUT
# Our scheme will now be 10th order (or less) in space.
order = 10

# Create our model passing it our 'upper' and 'lower' subdomains:
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=order, nbl=nbl, subdomains=(ur,lr), bcs="damp")
```

    Operator `initdamp` run in 0.01 s
    Operator `padfunc` run in 0.01 s

And re-define model related objects. Note that now our wave-field will be defined with `coefficients='symbolic'`.

```python
t0 = 0.  # Simulation starts a t=0
tn = 500.  # Simulation last 1 second (500 ms)
dt = 1.0  # Time step of 1.0ms

time_range = TimeAxis(start=t0, stop=tn, step=dt)

f0 = 0.025  # Source peak frequency is 25Hz (0.025 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 800.  # Depth is 800m

# New wave-field
u_DRP = TimeFunction(name="u_DRP", grid=model.grid, time_order=2, space_order=order, coefficients='symbolic')
```

We now create a stencil for each of our 'Upper' and 'Lower' subdomains defining different custom FD weights within each of these subdomains.

```python
# The underlying pde is the same in both subdomains
pde_DRP = model.m * u_DRP.dt2 - H + model.damp * u_DRP.dt

# Define our custom FD coefficients:
x, z = model.grid.dimensions
# Upper layer
weights_u = np.array([ 2.00462e-03, -1.63274e-02,  7.72781e-02,
                      -3.15476e-01,  1.77768e+00, -3.05033e+00,
                       1.77768e+00, -3.15476e-01,  7.72781e-02,
                      -1.63274e-02,  2.00462e-03])
# Lower layer
weights_l = np.array([  0.      ,  0.      ,  0.0274017,
                       -0.223818,  1.64875 , -2.90467,
                        1.64875 , -0.223818,  0.0274017,
                        0.      ,  0.       ])
# Create the Devito Coefficient objects:
ux_u_coeffs = Coefficient(2, u_DRP, x, weights_u/x.spacing**2)
uz_u_coeffs = Coefficient(2, u_DRP, z, weights_u/z.spacing**2)
ux_l_coeffs = Coefficient(2, u_DRP, x, weights_l/x.spacing**2)
uz_l_coeffs = Coefficient(2, u_DRP, z, weights_l/z.spacing**2)
# And the replacement rules:
coeffs_u = Substitutions(ux_u_coeffs,uz_u_coeffs)
coeffs_l = Substitutions(ux_l_coeffs,uz_l_coeffs)
# Create a stencil for each subdomain:
stencil_u = Eq(u_DRP.forward, solve(pde_DRP, u_DRP.forward).subs({H: u_DRP.laplace}),
               subdomain = model.grid.subdomains['upper'], coefficients=coeffs_u)
stencil_l = Eq(u_DRP.forward, solve(pde_DRP, u_DRP.forward).subs({H: u_DRP.laplace}),
               subdomain = model.grid.subdomains['lower'], coefficients=coeffs_l)

# Source term:
src_term = src.inject(field=u_DRP.forward, expr=src * dt**2 / model.m)

# Create the operator, incoporating both upper and lower stencils:
op = Operator([stencil_u, stencil_l] + src_term, subs=model.spacing_map)
```

And now execute the operator:

```python
#NBVAL_IGNORE_OUTPUT
op(time=time_range.num-1, dt=dt)
```

    Operator `Kernel` run in 0.03 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=0.02794600000000001, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section1', rank=None),
                         PerfEntry(time=0.00084200000000001, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

And plot the new results:

```python
fig = plt.figure(figsize=(14, 7))
ax1 = fig.add_subplot(111)
cont = ax1.contourf(X,Z,u_DRP.data[0,:,:], levels, cmap=cm.binary)
fig.colorbar(cont)
ax1.axis([0, Lx, 0, Lz])
ax1.set_xlabel('$x$')
ax1.set_ylabel('$z$')
ax1.set_title('$u_{DRP}(x,z,500)$')

plt.gca().invert_yaxis()
plt.show()
```

Finally, for comparison, lets plot the difference between the standard 20th order and optimized 10th order models:

```python
fig = plt.figure(figsize=(14, 7))
ax1 = fig.add_subplot(111)
cont = ax1.contourf(X,Z,abs(u_DRP.data[0,:,:]-u.data[0,:,:]), levels, cmap=cm.binary)
fig.colorbar(cont)
ax1.axis([0, Lx, 0, Lz])
ax1.set_xlabel('$x$')
ax1.set_ylabel('$z$')

plt.gca().invert_yaxis()
plt.show()
```

```python
#NBVAL_IGNORE_OUTPUT

# Wavefield norm checks
assert np.isclose(np.linalg.norm(u.data[-1]), 139.108, atol=0, rtol=1e-4)
assert np.isclose(np.linalg.norm(u_DRP.data[-1]), 83.636, atol=0, rtol=1e-4)
```