# TSSL Lab 1 - Autoregressive models

We load a few packages that are useful for solving this lab assignment.

```python
import pandas  # Loading data / handling data frames
import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model as lm  # Used for solving linear regression problems
from sklearn.neural_network import MLPRegressor # Used for NAR model

from tssltools_lab1 import acf, acfplot # Module available in LISAM - Used for plotting ACF
```

## 1.1 Loading, plotting and detrending data

In this lab we will build autoregressive models for a data set corresponding to the Global Mean Sea Level (GMSL) over the past few decades. The data is taken from https://climate.nasa.gov/vital-signs/sea-level/ and is available on LISAM in the file `sealevel.csv`.

**Q1**: Load the data and plot the GMSL versus time. How many observations are there in total in this data set?

_Hint:_ With pandas you can use the function `pandas.read_csv` to read the csv file into a data frame. Plotting the time series can be done using `pyplot`. Note that the sea level data is stored in the 'GMSL' column and the time when each data point was recorded is stored in the column 'Year'.

**A1**:

```python
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

gmsl = pd.read_csv('sealevel.csv')
print(f"Number of observations: {len(gmsl.index)}")

sns.lineplot(x='Year', y='GMSL', data=gmsl)
```

**Q2**: The data has a clear upward trend. Before fitting an AR model to this data need to remove this trend. Explain, using one or two sentences, why this is necessary.

**A2:**

Detrending the data means removing the global trend of the data over time (removing the mean function in this case). This is done by removing the mean function from the residuals. Benefit on this is that it enables discovering other aspects of the data like the auto correlations and dependences of the variables.

**Q3** Detrend the data following these steps:
1. Fit a straight line, $\mu_t=\theta_0 + \theta_1 u_t $ to the data based on the method of least squares. Here, $u_t$ is the time point when obervation $t$ was recorded.

    _Hint:_ You can use `lm.LinearRegression().fit(...)` from scikit-learn. Note that the inputs need to be passed as a 2D array.

    Before going on to the next step, plot your fitted line and the data in one figure.

2. Subtract the fitted line from $y_t$ for the whole data series and plot the deviations from the straight line.

**From now, we will use the detrended data in all parts of the lab.**

_Note:_ The GMSL data is recorded at regular time intervals, so that $u_{t+1} - u_t = $ const. Therefore, you can just as well use $t$ directly in the linear regression function if you prefer, $\mu_t=\theta_0 + \theta_1 t $.

**A3:**

```python
X = gmsl['Year'].to_numpy()
y = gmsl['GMSL'].to_numpy()

X = X.reshape(-1, 1)
#y = y.reshape(-1, 1)

model = lm.LinearRegression().fit(X,y)
```

```python
ypred = model.predict(X)

plt.plot(X, y,  color='blue')
plt.plot(X, ypred, color='red', linewidth=3)
```

```python
detrended = y - ypred

plt.plot(X,detrended, color='blue')
```

**Q4:** Split the (detrended) time series into training and validation sets. Use the values from the beginning up to the 700th time point (i.e. $y_t$ for $t=1$ to $t=700$) as your training data, and the rest of the values as your validation data. Plot the two data sets.

_Note:_ In the above, we have allowed ourselves to use all the available data (train + validation) when detrending. An alternative would be to use only the training data also when detrending the model. The latter approach is more suitable if, either:
* we view the linear detrending as part of the model choice. Perhaps we wish to compare different polynomial trend models, and evaluate their performance on the validation data, or
* we wish to use the second chunk of observations to estimate the performance of the final model on unseen data (in that case it is often referred to as "test data" instead of "validation data"), in which case we should not use these observations when fitting the model, including the detrending step.

In this laboration we consider the linear detrending as a predetermined preprocessing step and therefore allow ourselves to use the validation data when computing the linear trend.

**A4:**

```python
# splitting the feature
Xtrain = X[0:700,]
Xvalid = X[700:,]

# splitting the target
detrendedtrain = detrended[0:700,]
detrendedvalid = detrended[700:,]
```

## 1.2 Fit an autoregressive model
We will now fit an AR$(p)$ model to the training data for a given value of the model order $p$.

**Q5**: Create a function that fits an AR$(p)$ model for an arbitrary value of p. Use this function to fit a model of order $p=10$ to the training data and write out (or plot) the coefficients.

_Hint:_ Since fitting an AR model is essentially just a standard linear regression we can make use of `lm.LinearRegression().fit(...)` similarly to above. You may use the template below and simply fill in the missing code.

**A5:**

```python
def fit_ar(y, p):
    """Fits an AR(p) model. The loss function is the sum of squared errors from t=p+1 to t=n.

    :param y: array (n,), training data points
    :param p: int, AR model order
    :return theta: array (p,), learnt AR coefficients
    """

    # Number of training data points
    n = len(y) # <COMPLETE THIS LINE>

   # Construct the regression matrix
    Phi = np.zeros((n-p,p), dtype= int)

     #print("phi")
     #print(Phi)
     #print(Phi.shape)
    for j in range(p):
        Phi[:,j] = y[p-j:n-j] # <COMPLETE THIS LINE>

    # Drop the first p values from the target vector y
    yy = y[p:]  # yy = (y_{t+p+1}, ..., y_n)

    # Here we use fit_intercept=False since we do not want to include an intercept term in the AR model
    regr = lm.LinearRegression(fit_intercept=False)
    regr.fit(Phi,yy)

    return regr.coef_
```

```python
theta = fit_ar(detrendedtrain, 10)
theta
```

    array([ 1.04858020e+00,  1.03995740e-02,  6.87624028e-03,  6.22803308e-03,
            5.70699995e-03,  6.59110282e-03, -2.70953833e-03, -9.16897128e-03,
            6.66092353e-04, -7.84066761e-03])

**Q6:** Next, write a function that computes the one-step-ahead prediction of your fitted model. 'One-step-ahead' here means that in order to predict $y_t$ at $t=t_0$, we use the actual values of $y_t$ for $t<t_0$ from the data. Use your function to compute the predictions for both *training data* and *validation data*. Plot the predictions together with the data (you can plot both training and validation data in the same figure). Also plot the *residuals*.

_Hint:_ It is enought to call the predict function once, for both training and validation data at the same time.

**A6:**

```python
def predict_ar_1step(theta, y_target):
    """Predicts the value y_t for t = p+1, ..., n, for an AR(p) model, based on the data in y_target using
    one-step-ahead prediction.

    :param theta: array (p,), AR coefficients, theta=(a1,a2,...,ap).
    :param y_target: array (n,), the data points used to compute the predictions.
    :return y_pred: array (n-p,), the one-step predictions (\hat y_{p+1}, ...., \hat y_n)
    """

    n = len(y_target)
    p = len(theta)

    # Number of steps in prediction
    m = n-p
    y_pred = np.zeros(m)
    prevY = y_target[:10,]
    for i in range(m):
        # <COMPLETE THIS CODE BLOCK>
        #y_pred[i] = y_target[i+p:i].dot(theta) # <COMPLETE THIS LINE>
        prevY = prevY[i:p+i]
        y_pred[i] = prevY.dot(theta)
        prevY = np.append(prevY , y_pred)

    return y_pred
```

```python
pred_train = predict_ar_1step(theta, detrendedtrain)
pred_valid = predict_ar_1step(theta, detrendedvalid)
```

```python
pred_valid.shape
```

    (287,)

```python
plt.plot(Xtrain[10:,], detrendedtrain[10:,],  color='blue')
plt.plot(Xtrain[10:,], pred_train, color='red')
```

```python
plt.plot(Xvalid[10:,], detrendedvalid[10:,],  color='blue')
plt.plot(Xvalid[10:,], pred_valid, color='red')
```

```python
pred_valid
```

    array([ -3.1360453 ,  -5.13169701,  -6.02434681,  -8.64803883,
            -3.32547717,  -6.40300014,  -5.43193911,  -9.14305106,
            -6.77966579,  -9.64969393,  -3.32853436,  -5.48956174,
            -6.41328221,  -9.17417627,  -3.62319396,  -6.85667756,
            -5.79996074,  -9.7194242 ,  -7.21849221, -10.18513903,
            -3.53454887,  -5.87249929,  -6.82839194,  -9.73422364,
            -3.94406179,  -7.34034545,  -6.19206856, -10.33103593,
            -7.68422883, -10.75111814,  -3.75511776,  -6.28227117,
            -7.27146231, -10.33038623,  -4.28970712,  -7.85589204,
            -6.60979393, -10.98001137,  -8.17852996, -11.3494737 ,
            -3.99134905,  -6.72076084,  -7.74439965, -10.96501006,
            -4.66186302,  -8.40532135,  -7.05476508, -11.6686053 ,
            -8.70315551, -11.98216617,  -4.24443569,  -7.18998185,
            -8.24923807, -11.64059071,  -5.0623759 ,  -8.99076027,
            -7.52871332, -12.39921028,  -9.25997793, -12.65128186,
            -4.51566188,  -7.69208663,  -8.78814786, -12.35978256,
            -5.49321264,  -9.61446611,  -8.03347973, -13.17436539,
            -9.85098976, -13.35904127,  -4.80640979,  -8.22937575,
            -9.36344427, -13.12540872,  -5.95646819, -10.27883465,
            -8.57102227, -13.99676538, -10.47831161, -14.10780804,
            -5.11816678,  -8.8043078 ,  -9.97759696, -13.94047146,
            -6.45437367, -10.98640864,  -9.14342339, -14.8692706 ,
           -11.14420071, -14.90009846,  -5.45253311,  -9.41951   ,
           -10.63323998, -14.8081634 ,  -6.98930491, -11.73988689,
            -9.75289808, -15.79491744, -11.85106004, -15.73859161,
            -5.81123021, -10.07778934, -11.3331824 , -15.73187928,
            -7.56379158, -12.54213388, -10.40180253, -16.77692951,
           -12.60144809, -16.62614019,  -6.19610949, -10.78214454,
           -12.08041963, -16.71522847,  -8.18052694, -13.39619005,
           -11.09264338, -17.81872952, -13.39808925, -17.56578205,
            -6.60916179, -11.53577882, -12.8781454 , -17.76204835,
            -8.84237807, -14.30528266, -11.82808751, -18.92395199,
           -14.24388494, -18.56075252,  -7.05252751, -12.34211334,
           -13.72976461, -18.87641837,  -9.55239687, -15.27283745,
           -12.61097257, -20.09645668, -15.14192546, -19.61449753,
            -7.52850734, -13.20480168, -14.63890683, -20.06267508,
           -10.31383173, -16.30249094, -13.44431819, -21.34034304,
           -16.09550266, -20.73068762,  -8.03957391, -14.12774512,
           -15.6094408 , -21.32542804, -11.13013988, -17.39810357,
           -14.33133793, -22.65996547, -17.10812351, -21.91323282,
            -8.58838404, -15.11510894, -16.64548974, -22.66957668,
           -12.00500062, -18.56377366, -15.27545203, -24.05994968,
           -18.18352449, -23.16629862,  -9.17779195, -16.17133983,
           -17.75144768, -24.10032831, -12.9423293 , -19.80385231,
           -16.2803011 , -25.54521004, -19.32568701, -24.49432291,
            -9.8108634 , -17.30118427, -18.93199682, -25.62321711,
           -13.94629223, -21.12295922, -17.34976062, -27.12096814,
           -20.5388539 , -25.90203403, -10.49089067, -18.50970825,
           -20.19212592, -27.24412436, -15.02132256, -22.52599949,
           -18.48795652, -28.79277249, -21.82754698, -27.39447011,
           -11.22140872, -19.80231811, -21.53714997, -28.96929993,
           -16.17213715, -24.0181816 , -19.69928172, -30.56651957,
           -23.19658577, -28.97699954, -12.00621232, -21.18478278,
           -22.97273094, -30.80538505, -17.40375451, -25.60503649,
           -20.98841387, -32.44847621, -24.65110753, -30.65534293,
           -12.84937438, -22.66325748, -24.50489999, -32.75943655,
           -18.72151388, -27.29243788, -22.36033428, -34.44530348,
           -26.19658864, -32.4355964 , -13.75526563, -24.2443088 ,
           -26.14008096, -34.83895252, -20.13109561, -29.0866239 ,
           -23.82034807, -36.56408212, -27.83886736, -34.32425648,
           -14.72857542, -25.9349415 , -27.88511544, -37.05189965,
           -21.63854273, -30.99422015, -25.37410576, -38.81233961,
           -29.58416821, -36.32824657, -15.77433414, -27.74262694,
           -29.7472893 , -39.40674219, -23.25028399, -33.02226426,
           -27.02762628, -41.19807908, -31.43912787, -38.4549452 ,
           -16.89793702, -29.6753334 , -31.73436104, -41.91247277,
           -24.97315838, -35.17823201, -28.78732155, -43.72981005,
           -33.41082294, -40.71221611, -18.10516957, -31.74155819,
           -33.85459179, -44.57864506, -26.81444116, -37.47006518,
           -30.66002273, -46.41658122, -35.50679944, -43.10844028,
           -19.40223475, -33.95036196, -36.11677726, -47.41540854,
           -28.78187166, -39.90620122, -32.65300827])

```python
plt.plot(Xvalid[700:,], detrendedvalid[700:,],  color='blue')
plt.plot(Xvalid[700:,], pred_valid, color='red', linewidth=3)
```

**Q7:** Compute and plot the autocorrelation function (ACF) of the *residuals* only for the *validation data*. What conclusions can you draw from the ACF plot?

_Hint:_ You can use the function `acfplot` from the `tssltools` module, available on the course web page.

**A7:**

```python
help(acfplot)
```

    Help on function acfplot in module tssltools_lab1:

    acfplot(x, lags=None, conf=0.95)
        Plots the empirical autocorralation function.

        :param x: array (n,), sequence of data points
        :param lags: int, maximum lag to compute the ACF for. If None, this is set to n. Default is None.
        :param conf: float, number in the interval [0,1] which specifies the confidence level (based on a central limit
                     theorem under a white noise assumption) for two dashed lines drawn in the plot. Default is 0.95.
        :return:

## 1.3 Model validation and order selection
Above we set the model order $p=10$ quite arbitrarily. In this section we will try to find an appropriate order by validation.

**Q8**: Write a loop in which AR-models of orders from $p=2$ to $p=150$ are fitted to the data above. Plot the training and validation mean-squared errors for the one-step-ahead predictions versus the model order.

Based on your results:
- What is the main difference between the changes in training error and validation error as the order increases?
- Based on these results, which model order would you suggest to use and why?

_Note:_ There is no obvious "correct answer" to the second question, but you still need to pick an order an motivate your choice!

**A8:**

```python

```

**Q9:** Based on the chosen model order, compute the residuals of the one-step-ahead predictions on the *validation data*. Plot the autocorrelation function of the residuals. What conclusions can you draw? Compare to the ACF plot generated above for p=10.

```python

```

## 1.4 Long-range predictions
So far we have only considered one-step-ahead predictions. However, in many practical applications it is of interest to use the model to predict further into the future. For intance, for the sea level data studied in this laboration, it is more interesting to predict the level one year from now, and not just 10 days ahead (10 days = 1 time step in this data).

**Q10**:
Write a function that simulates the value of an AR($p$) model $m$ steps into the future, conditionally on an initial sequence of data points. Specifically, given $y_{1:n}$ with $n\geq p$ the function/code should predict the values

\begin{align}
    \hat y_{t|n} &= \mathbb{E}[y_{t} | y_{1:n}], & t&=n+1,\dots,n+m.
\end{align}

Use this to predict the values for the validation data ($y_{701:997}$) conditionally on the training data ($y_{1:700}$) and plot the result.

_Hint:_ Use the pseudo-code derived at the first pen-and-paper session.

**A10:**

```python
def simulate_ar(y, theta, m):
    """Simulates an AR(p) model for m steps, with initial condition given by the last p values of y

    :param y: array (n,) with n>=p. The last p values are used to initialize the simulation.
    :param theta: array (p,). AR model parameters,
    :param m: int, number of time steps to simulate the model for.
    """

    p = len(theta)
    y_sim = np.zeros(m)
    phi = np.flip(y[-p:].copy()) # (y_{n-1}, ..., y_{n-p})^T - note that y[ntrain-1] is the last training data point

    for i in range(m):
        y_sim[i] = ... # <COMPLETE THIS LINE>
        # <COMPLETE THIS CODE BLOCK>

    return y_sim

```

```python

```

**Q11:** Using the same function as above, try to simulate the process for a large number of time steps (say, $m=2000$). You should see that the predicted values eventually converge to a constant prediction of zero. Is this something that you would expect to see in general? Explain the result.

**A11:**

```python

```

## 1.5 Nonlinear AR model
 In this part, we switch to a nonlinear autoregressive (NAR) model, which is based on a feedforward neural network. This means that in this model the recursive equation for making predictions is still in the form $\hat y_t=f_\theta(y_{t-1},...,y_{t-p})$, but this time $f$ is a nonlinear function learned by the neural network. Fortunately almost all of the work for implementing the neural network and training it is handled by the `scikit-learn` package with a few lines of code, and we just need to choose the right structure, and prepare the input-output data.

**Q12**: Construct a NAR($p$) model with a feedforward (MLP) network, by using the `MLPRegressor` class from `scikit-learn`. Set $p$ to the same value as you chose for the linear AR model above. Initially, you can use an MLP with a single hidden layer consisting of 10 hidden neurons.
Train it using the same training data as above and plot the one-step-ahead predictions as well as the residuals, on both the training and validation data.

_Hint:_ You will need the methods `fit` and `predict` of `MLPRegressor`. Read the user guide of `scikit-learn` for more details. Recall that a NAR model is conceptuall very similar to an AR model, so you can reuse part of the code from above.

**A12:**

```python

```

```python

```

**Q13:** Try to expirement with different choices for the hyperparameters of the network (e.g. number of hidden layers and units per layer, activation function, etc.) and the optimizer (e.g. `solver` and `max_iter`).

Are you satisfied with the results? Why/why not? Discuss what the limitations of this approach might be.

**A13:**

```python

```

```python

```


NOTE: Videos and graphs do not display immediately on all devices.  If this is the case for you, click the "Cell" menu option, and select "Run All".  If you still do not see the videos/graphs, try a different browser.  If that also does not work, please let your instructor know by email.

# Definite Integrals for Future Value

In this notebook, we concern ourselves with the determination of the future value of a quantity of interest (such as the amount of water in a reservoir, the amount of pollution in a lake, the number of indivuals in a population, etc.).

In Sections 1-3, we return to a very familiar concept, namely the calculation of net or cumulative change of a quantity by integrating the known rate of change of that quantity with respect to time.  This readily leads to the determination of the future value.

In Sections 4-6, we introduce a new concept, namely the interaction of a renewal function (representing the rate at which the quantity increases) and a survival function (representing the proportion of a quantity that survives for a certain period of time).  We use the "slicing and integrating method" to develop a formula for future value.

### 1. An Old Idea: Interpreting a Definite Integral in Terms of Net Change or Cumulative Change

From the Fundamental Theorem of Calculus, we know that
$$\int_a^b f(x) \ dx = F(b) - F(a),$$
where $F$ is any antiderivative of $f(x)$.

A common first application of the Fundamental Theorem of Calculus is in the determination of displacement of an object with given velocity function.  Suppose the position of an object as a function of time is given by $s(t)$.  The velocity of the object as a function of time is $v(t) = s'(t)$, that is, $v$ is an antiderivative of $s$.  Then
$$\int_a^b v(t) \ dt = s(b) - s(a).$$
We interpret the latter as the displacement of the object from $t=a$ to $t=b$.

If we let $a=0$ and $b=T$, then it immediately follows that
$$s(T) = s(0) + \int_0^T v(t) \ dt.$$

In words, we can interpret the latter as "the position at future time $T$ is the current position (at time 0) plus the net or cumulative change in position".

### 2. Generalization: Future Value from Net Change or Cumulative Change

We can generalize the above ideas to other quantities, such as the amount of water in a reservoir, the amount of pollution in a lake, the number of indivuals in a population, etc.

Suppose a quantity $Q$ changes over time at a known rate $Q'(t)$. Then the net (cumulative) change in Q between $t=a$ and $t=b$ is $$Q(b) - Q(a) = \int_a^b Q'(t) \ dt,$$
and the future value of $Q$ at time $T$ is $$Q(T) = Q(0) + \int_0^T Q'(t) \ dt.$$

### 3. Example of Future Value from Net Change or Cumulative Change

[Credit: W. Briggs and L. Cochran, Calculus: Early Transcendentals, 2011, Pearson Educationc Inc. ]

A culture of cells in a lab has a population of 100 cells when nutrients are added at time $t=0$.  Suppose the population $N(t)$ increases at a rate given by $\displaystyle N'(t) = 90 e^{-0.1t}$ cells/hr.

Then the population at future time $T$ is
\begin{align}
N(T)
& = N(0) + \int_0^T N'(t) \ dt \\
& = 100 + \int_0^T 90 e^{-0.1t} \ dt \\
& = 100 + \left. \left( \frac{90}{-0.1} e^{-0.1t} \right) \right|_0^T \\
& = 1000 - 900 e^{-0.1T}.
\end{align}

Note that this population increases, but at a decreasing rate.  As $t \to \infty$, the population size approaches 1000 cells.

### 4. A New Idea: Future Value from Survival and Renewal

For some quantities of interest, the effective rate of change is not known, but there is information about other processes (which we will refer to as survival and renewal) affecting change that allow us to determine the future value of the quantity.

Renewal refers to the rate at which the quantity Q increases.  Let $R(t)$ be the renewal function.
- If $Q$ represents the size of a population, then $R(t)$ represents the rate at which new members are added to the population at time $t$.
- If $Q$ represents the amount of a drug in the body, then $R(t)$ represents the amount of drug administered at time $t$.

Survival refers to the proportion of the quantity $Q$ that survives or remains.  Let $S(t)$ be the survival function.
- If $Q$ represents the size of a population, then $S(t)$ represents the proportion of the population that survives at least $t$ time units from now.
- If $Q$ represents the amount of a drug in the body, then $S(t)$ represents the proportion of the drug that remains in the body at least $t$ time units from now.
>For example, if $S(10) = 0.7$, then 70% of the current quantity $Q$ survives/remains after $10$ time units.

As before in Section 2, we are interested in determining $Q(T)$, the value of $Q$ at future time $T$.  We need to do this from our knowledge of $Q(0)$ and the renewal and survival functions, $R(t)$ and $S(t)$, respectively.

> It is key to realize that the renewal and survival processes referred to here are not independent; they interact and work simultaneously.  In general, more of a small quantity added late in time will survive/remain at future time $T$ than a small quantity added earlier in time.

### 5. Slicing Time and Integrating to determine Future Value from Survival and Renewal

Let's determine first what happens to the initial quantity $Q(0)$.
- The proportion of $Q(0)$ that survives/remains at time $T$ is $S(T)$.
- The amount of $Q(0)$ that survives/remains at time $T$ is the product, that is,
$$S(T) \cdot Q(0).$$

Now focus on what happens during the time interval $[0,T]$.  We slice time, focus on a small infinitessimal time period $dt$ at generic time $t$, as illustrated below, and integrate.

- The small quantity added during this infinitessimal time period $dt$ is "rate of renewal times time", or $R(t) \ dt.$
- The length of the time interval $[t,T]$ is $T-t$.  Thus, the proportion of the small quantity added that survives/remains at time $T$ is $S(T-t).$
- The amount of the small quantity that is added at time $t$ AND that survives/remains at time $T$ is the product of the above, that is, $$dQ = S(T-t) \cdot R(t) \ dt.$$
- The total quantity that is added during the time period from $t=0$ to $t=T$ AND that survives/remains at time $T$ is $$\int dQ = \int_0^T S(T-t) \cdot R(t) \ dt.$$

Putting everything together, we have that $Q(T)$ is the initial amount that survives/remains at time $T$ PLUS the amount that has been added during the time interval $[0,T]$ and survives/remains at time $T$, namely
\begin{align}
Q(T)
& = S(T) \cdot Q(0) + \int dQ \\
& = S(T) \cdot Q(0) + \int_0^T S(T-t) \cdot R(t) \ dt.
\end{align}

### 6. Example of Future Value from Survival and Renewal

[Credit: J. Stewart and T. Day, Biocalculus: Calculus for the Life Sciences, 2015, Cengage Learning.]

Consider a lake with trout.  There currently are 5600 trout in the lake.  The trout are reproducing at the rate $\displaystyle R(t) = 720 e^{0.1t}$ trout/year.  However, polution is killing many of the trout; the proportion that survive after $t$ years is given by $\displaystyle S(t) = e^{-0.2t}$.  How many trout will be in the lake 10 years from now?

Let $P(t)$ be the number of trout in the lake at time $t$, measured in years.

We are given $P(0) = 5600$ trout, and $T = 10$ years.  Then
\begin{align}
P(10)
& = S(10) \cdot P(0) + \int_0^{10} S(10-t) \cdot R(t) \ dt \\
& = e^{-0.2(10)} \cdot 5600 + \int_0^{10} e^{-0.2(10-t)} \cdot 720 e^{0.1t} \ dt \\
& = 5600 e^{-2} + 720 \int_0^{10} e^{-2+0.2t+0.1t} \ dt \\
& = 5600 e^{-2} + 720 e^{-2} \int_0^{10} e^{0.3t} \ dt \\
& = 5600 e^{-2} + 720 e^{-2} \left. \left( \frac{ e^{0.3t} }{0.3} \right) \right|_0^{10} \\
& = 5600 e^{-2} + \frac{ 720 e^{-2} }{0.3} \left( e^{3} - 1 \right) \\
& = 5600 e^{-2} + 2400 \left( e - e^{-2} \right) \\
& \approx 6956.95.
\end{align}
Thus, we expect the number of trout in the lake 10 years from now to be about 6957.

### 7. Summary

Future Value from Net Change or Cumulative Change:

Suppose a quantity $Q$ changes over time at a known rate $Q'(t)$.
- The net (cumulative) change in Q between $t=a$ and $t=b$ is $$Q(b) - Q(a) = \int_a^b Q'(t) \ dt.$$
- The future value of $Q$ at time $T$ is $$Q(T) = Q(0) + \int_0^T Q'(t) \ dt.$$

Future Value from Survival and Renewal:

Suppose a quantity $Q(t)$ has initial value $Q(0)$, renewal function $R(t)$ that represents the rate at which the quantity increases, and survival function $S(t)$ that represents the proportion of the quantity that survives/remains after $t$ years.
- The future value of $Q$ at time $T$ is
$$Q(T) = S(T) \cdot Q(0) + \int_0^T S(T-t) \cdot R(t) \ dt.$$

### 8. Further Study

Please refer to Section 6.3 in the textbook for additional treatment of this topic.

### 9. Don't Forget

Don't forget to return to eClass to complete the pre-class quiz.

In the pre-class quiz feedback question, please let me know if this notebook would benefit from one or two videos.  If so, which section would you like to see become a video?  The derivation of Future Value in Section 5, or the example in Section 6, or both?  Thank you!


# Задания

* $f(x)=x^3+3x^2-72x+1$ Найти производную первого порядка и нарисовать график
* $f(x)=ln(2x-1)$ Найти производную 10 порядка и нарисовать график
* $f(x,y)=x^2+y^2$ Найти производную по х, по y и нарисовать графики

PS Видео решения дз не будет, смысла не вижу

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from sympy import *

def showPlot(x_start, x_max, step, callback, title='Function'):
    x = np.arange(x_start, x_max, step)
    y = callback(x)

    plt.plot(x, y)
    plt.xlabel('X')
    plt.ylabel('Y')
    plt.title(title)
    plt.grid(True)
    plt.show()

def value_of_expr(x_vector, symbol, expr):
    y = np.empty_like(x_vector)
    for index, element  in enumerate(x_vector):
        y[index] = (expr.subs(symbol, element))
    return y

def value_of_expr_3d(x_matrix, y_matrix, x_symbol, y_symbol, expr):
    z = np.empty_like(x_matrix)
    for i_1 in range(0, z.shape[0]):
        for i_2 in range(0, z.shape[0]):
            z[i_1, i_2] = expr.subs({x_symbol: x_matrix[i_1, i_2], y_symbol: y_matrix[i_1, i_2]})
    return z
```

```python
init_printing()
x,y = symbols('x y')
```

```python
e = x**3+3*x**2-72*x+1
dx = diff(e, x)
showPlot(-10, 10, 0.1, lambda i: value_of_expr(i,x,e), 'Origin')
showPlot(-10, 10, 0.1, lambda i: value_of_expr(i,x,dx), 'Diff')
```

```python
e = log(2*x-1)
dx = diff(e, x, 10)
showPlot(4, 10, 0.1, lambda i: value_of_expr(i,x,e), 'Origin')
showPlot(4, 10, 0.1, lambda i: value_of_expr(i,x,dx), 'Diff')
```

```python
from mpl_toolkits.mplot3d import axes3d, Axes3D

fig = plt.figure()
ax = Axes3D(fig)

e = x**2 + y**2

# Make data.
X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
Z = value_of_expr_3d(X, Y, x, y, e)

surf = ax.plot_surface(X, Y, Z, linewidth=0)

dx = diff(e,x)
Z = value_of_expr_3d(X, Y, x, y, dx)
surf = ax.plot_surface(X, Y, Z, linewidth=0, color='g')

dy = diff(e,y)
Z = value_of_expr_3d(X, Y, x, y, dy)
surf = ax.plot_surface(X, Y, Z, linewidth=0, color='r')

# Customize the z axis.
ax.set_zlim(0, 20)
```

```python
dx
```

```python
dy
```

```python

```


# 01 - Introduction to seismic modelling

This notebook is the first in a series of tutorials highlighting various aspects of seismic inversion based on Devito operators. In this first example we aim to highlight the core ideas behind seismic modelling, where we create a numerical model that captures the processes involved in a seismic survey. This forward model will then form the basis for further tutorials on the implementation of inversion processes using Devito operators.

## Modelling workflow

The core process we are aiming to model is a seismic survey, which consists of two main components:

- **Source** - A source is positioned at a single or a few physical locations where artificial pressure is injected into the domain we want to model. In the case of land survey, it is usually dynamite blowing up at a given location, or a vibroseis (a vibrating engine generating continuous sound waves). For a marine survey, the source is an air gun sending a bubble of compressed air into the water that will expand and generate a seismic wave.
- **Receiver** - A set of microphones or hydrophones are used to measure the resulting wave and create a set of measurements called a *Shot Record*. These measurements are recorded at multiple locations, and usually at the surface of the domain or at the bottom of the ocean in some marine cases.

In order to create a numerical model of a seismic survey, we need to solve the wave equation and implement source and receiver interpolation to inject the source and record the seismic wave at sparse point locations in the grid.

## The acoustic seismic wave equation
The acoustic wave equation for the square slowness $m$, defined as $m=\frac{1}{c^2}$, where $c$ is the speed of sound in the given physical media, and a source $q$ is given by:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) = q \ \text{in } \Omega \\
 &u(.,t=0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

with the zero initial conditions to guarantee unicity of the solution.
The boundary conditions are Dirichlet conditions:
\begin{equation}
 u(x,t)|_\delta\Omega = 0
\end{equation}

where $\delta\Omega$ is the surface of the boundary of the model $\Omega$.

# Finite domains

The last piece of the puzzle is the computational limitation. In the field, the seismic wave propagates in every direction to an "infinite" distance. However, solving the wave equation in a mathematically/discrete infinite domain is not feasible. In order to compensate, Absorbing Boundary Conditions (ABC) or Perfectly Matched Layers (PML) are required to mimic an infinite domain. These two methods allow to approximate an infinite media by damping and absorbing the waves at the limit of the domain to avoid reflections.

The simplest of these methods is the absorbing damping mask. The core idea is to extend the physical domain and to add a Sponge mask in this extension that will absorb the incident waves. The acoustic wave equation with this damping mask can be rewritten as:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) + \eta \frac{d u(x,t)}{dt}=q  \ \text{in } \Omega \\
 &u(.,0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

where $\eta$ is the damping mask equal to $0$ inside the physical domain and increasing inside the sponge layer. Multiple choice of profile can be chosen for $\eta$ from linear to exponential.

# Seismic modelling with devito

We describe here a step by step setup of seismic modelling with Devito in a simple 2D case. We will create a physical model of our domain and define a single source and an according set of receivers to model for the forward model. But first, we initialize some basic utilities.

```python
#NBVAL_IGNORE_OUTPUT
# Adding ignore due to (probably an np notebook magic) bug
import numpy as np
%matplotlib inline
```

## Define the physical problem

The first step is to define the physical model:

- What are the physical dimensions of interest
- What is the velocity profile of this physical domain

We will create a simple velocity model here by hand for demonstration purposes. This model essentially consists of two layers, each with a different velocity: $1.5km/s$ in the top layer and $2.5km/s$ in the bottom layer. We will use this simple model a lot in the following tutorials, so we will rely on a utility function to create it again later.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Model, plot_velocity

# Define a physical size
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # What is the location of the top left corner. This is necessary to define
# the absolute location of the source and receivers

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:, :51] = 1.5
v[:, 51:] = 2.5

# With the velocity and model size defined, we can create the seismic model that
# encapsulates this properties. We also define the size of the absorbing layer as 10 grid points
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=2, nbl=10, bcs="damp")

plot_velocity(model)
```

# Acquisition geometry

To fully define our problem setup we also need to define the source that injects the wave to model and the set of receiver locations at which to sample the wavefield. The source time signature will be modelled using a Ricker wavelet defined as

\begin{equation}
  q(t) = (1-2\pi^2 f_0^2 (t - \frac{1}{f_0})^2 )e^{- \pi^2 f_0^2 (t - \frac{1}{f_0})}
\end{equation}

To fully define the source signature we first need to define the time duration for our model and the timestep size, which is dictated by the CFL condition and our grid spacing. Luckily, our `Model` utility provides us with the critical timestep size, so we can fully discretize our model time axis as an array:

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 1000.  # Simulation last 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing

time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

The source is positioned at a $20m$ depth and at the middle of the $x$ axis ($x_{src}=500m$), with a peak wavelet frequency of $10Hz$.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import RickerSource

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

# First, position source centrally in all dimensions, then set depth
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 20.  # Depth is 20m

# We can plot the time signature to see the wavelet
src.show()
```

Similarly to our source object, we can now define our receiver geometry as a symbol of type `Receiver`. It is worth noting here that both utility classes, `RickerSource` and `Receiver` are thin wrappers around the Devito's `SparseTimeFunction` type, which encapsulates sparse point data and allows us to inject and interpolate values into and out of the computational grid. As we have already seen, both types provide a `.coordinates` property to define the position within the domain of all points encapsulated by that symbol.

In this example we will position receivers at the same depth as the source, every $10m$ along the x axis. The `rec.data` property will be initialized, but left empty, as we will compute the receiver readings during the simulation.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Receiver

# Create symbol for 101 receivers
rec = Receiver(name='rec', grid=model.grid, npoint=101, time_range=time_range)

# Prescribe even spacing for receivers along the x-axis
rec.coordinates.data[:, 0] = np.linspace(0, model.domain_size[0], num=101)
rec.coordinates.data[:, 1] = 20.  # Depth is 20m

# We can now show the source and receivers within our domain:
# Red dot: Source location
# Green dots: Receiver locations (every 4th point)
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])
```

# Finite-difference discretization

Devito is a finite-difference DSL that solves the discretized wave-equation on a Cartesian grid. The finite-difference approximation is derived from Taylor expansions of the continuous field after removing the error term.

## Time discretization

We only consider the second order time discretization for now. From the Taylor expansion, the second order discrete approximation of the second order time derivative is:
\begin{equation}
\begin{aligned}
 \frac{d^2 u(x,t)}{dt^2} = \frac{\mathbf{u}(\mathbf{x},\mathbf{t+\Delta t}) - 2 \mathbf{u}(\mathbf{x},\mathbf{t}) + \mathbf{u}(\mathbf{x},\mathbf{t-\Delta t})}{\mathbf{\Delta t}^2} + O(\mathbf{\Delta t}^2).
\end{aligned}
\end{equation}

where $\mathbf{u}$ is the discrete wavefield, $\mathbf{\Delta t}$ is the discrete
time-step (distance between two consecutive discrete time points) and $O(\mathbf{\Delta
  t}^2)$ is the discretization error term. The discretized approximation of the
second order time derivative is then given by dropping the error term. This derivative is represented in Devito by `u.dt2` where u is a `TimeFunction` object.

## Spatial discretization

We define the discrete Laplacian as the sum of the second order spatial
derivatives in the three dimensions:
\begin{equation}
\begin{aligned}
\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t})= \sum_{j=1}^{j=\frac{k}{2}} \Bigg[\alpha_j \Bigg(&
\mathbf{u}(\mathbf{x+jdx},\mathbf{y},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x-jdx},\mathbf{y},\mathbf{z},\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y+jdy},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y-jdy},\mathbf{z}\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z+jdz},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z-jdz},\mathbf{t})\Bigg) \Bigg] + \\
&3\alpha_0 \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}).
\end{aligned}
\end{equation}

This derivative is represented in Devito by `u.laplace` where u is a `TimeFunction` object.

## Wave equation

With the space and time discretization defined, we can fully discretize the wave-equation with the combination of time and space discretizations and obtain the following second order in time and $k^{th}$ order in space discrete stencil to update one grid point at position $\mathbf{x}, \mathbf{y},\mathbf{z}$ at time $\mathbf{t}$, i.e.
\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

```python
# In order to represent the wavefield u and the square slowness we need symbolic objects
# corresponding to time-space-varying field (u, TimeFunction) and
# space-varying field (m, Function)
from devito import TimeFunction

# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=2)

# We can now write the PDE
pde = model.m * u.dt2 - u.laplace + model.damp * u.dt

# The PDE representation is as on paper
pde
```

$\displaystyle \operatorname{damp}{\left(x,y \right)} \frac{\partial}{\partial t} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial x^{2}} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial y^{2}} u{\left(t,x,y \right)} + \frac{\frac{\partial^{2}}{\partial t^{2}} u{\left(t,x,y \right)}}{\operatorname{vp}^{2}{\left(x,y \right)}}$

```python
# This discrete PDE can be solved in a time-marching way updating u(t+dt) from the previous time step
# Devito as a shortcut for u(t+dt) which is u.forward. We can then rewrite the PDE as
# a time marching updating equation known as a stencil using customized SymPy functions
from devito import Eq, solve

stencil = Eq(u.forward, solve(pde, u.forward))
stencil
```

$\displaystyle u{\left(t + dt,x,y \right)} = \frac{- \frac{- \frac{2.0 u{\left(t,x,y \right)}}{dt^{2}} + \frac{u{\left(t - dt,x,y \right)}}{dt^{2}}}{\operatorname{vp}^{2}{\left(x,y \right)}} + \frac{\partial^{2}}{\partial x^{2}} u{\left(t,x,y \right)} + \frac{\partial^{2}}{\partial y^{2}} u{\left(t,x,y \right)} + \frac{\operatorname{damp}{\left(x,y \right)} u{\left(t,x,y \right)}}{dt}}{\frac{\operatorname{damp}{\left(x,y \right)}}{dt} + \frac{1}{dt^{2} \operatorname{vp}^{2}{\left(x,y \right)}}}$

# Source injection and receiver interpolation

With a numerical scheme to solve the homogenous wave equation, we need to add the source to introduce seismic waves and to implement the measurement operator, and interpolation operator. This operation is linked to the discrete scheme and needs to be done at the proper time step. The semi-discretized in time wave equation with a source reads:

\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

It shows that in order to update $\mathbf{u}$ at time $\mathbf{t+\Delta t}$ we have to inject the value of the source term $\mathbf{q}$ of time $\mathbf{t}$. In Devito, it corresponds the update of $u$ at index $t+1$ (t = time implicitly) with the source of time $t$.
On the receiver side, the problem is either as it only requires to record the data at the given time step $t$ for the receiver at time $time=t$.

```python
# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)

# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=u.forward)
```

```python
print(src_term)
print('---------------')
print(rec_term)
```

    Injection(5.99760042078401*src(time, p_src)*vp(x, y)**2 into u(t + dt, x, y))
    ---------------
    Interpolation(u(t + dt, x, y) into rec(time, p_rec))

# Devito operator and solve
After constructing all the necessary expressions for updating the wavefield, injecting the source term and interpolating onto the receiver points, we can now create the Devito operator that will generate the C code at runtime. When creating the operator, Devito's two optimization engines will log which performance optimizations have been performed:
* **DSE:** The Devito Symbolics Engine will attempt to reduce the number of operations required by the kernel.
* **DLE:** The Devito Loop Engine will perform various loop-level optimizations to improve runtime performance.

**Note**: The argument `subs=model.spacing_map` causes the operator to substitute values for our current grid spacing into the expressions before code generation. This reduces the number of floating point operations executed by the kernel by pre-evaluating certain coefficients.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

op = Operator([stencil] + src_term + rec_term, subs=model.spacing_map)
```

Now we can execute the create operator for a number of timesteps. We specify the number of timesteps to compute with the keyword `time` and the timestep size with `dt`.

```python
#NBVAL_IGNORE_OUTPUT
op(time=time_range.num-1, dt=model.critical_dt)
```

    Operator `Kernel` ran in 0.04 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=0.03294000000000005, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section1', rank=None),
                         PerfEntry(time=4.899999999999996e-05, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section2', rank=None),
                         PerfEntry(time=0.0011690000000000003, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

After running our operator kernel, the data associated with the receiver symbol `rec.data` has now been populated due to the interpolation expression we inserted into the operator. This allows us the visualize the shot record:

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

plot_shotrecord(rec.data, model, t0, tn)
```

```python
assert np.isclose(np.linalg.norm(rec.data), 370, rtol=1)
```


# Multidimensional Scaling

In 12 hours I'll be finishing my last exam for my masters! This one is for [Data Mining and Exploration](https://www.inf.ed.ac.uk/teaching/courses/dme/). One topic we covered was *Multidimensional Scaling* (MDS), which turns out to require a lot of the neat things we learned in class. MDS methods take similarity between pairs of points, such as distance, and uses them to estimate where the points are in space while trying to preserve their pairwise similarity. This method is useful if you want to visualize points but all you have is a measure of something like distance.

One way of doing MDS is to assume the similarities are Euclidean distances. While other MDS techniques require iterative optimization, the Euclidean-distance-based approach has a closed-form solution based on PCA.

To summarize, MDS takes in $\Delta$, an $N \times N$ symmetric matrix of distances, and returns a $K \times N$ matrix of embedded data points, where $N$ is the number of examples in our dataset, and $K$ is a user-defined number of dimensions that the points should be in.

In order to demonstrate MDS, I first generate fake underlying data points and then compute distances from them. Then I'll compute a few other forms that will be useful for comparision: the *Gram matrix* and a centered data matrix using a *centering matrix*.
Then I'll apply MDS to the generated distance matrix. At the end, I apply MDS to a dataset of distances between North American cities resulting in the plot below. It might be squashed and upside-down, but it almost looks like a map of North America!

```python
import numpy as np
import matplotlib.pyplot as plt

from sklearn.metrics.pairwise import euclidean_distances
```

```python
# helper functions you can skip over :D
def hide_ticks(plot):
    plot.axes.get_xaxis().set_visible(False)
    plot.axes.get_yaxis().set_visible(False)

SAVE = True
def maybe_save_plot(filename):
    if SAVE:
        plt.tight_layout()
        plt.savefig('images/' + filename, bbox_inches="tight")
```

## Data generation

Usually the point of using MDS is you don't have the underlying points. However, I'll generate some points so I can get a distance matrix from them as well as compare intermediate forms. In this case, the underlying data is in a much higher dimension.

I'll start with some Gaussian data that is rotated and shifted. I also define some constants:

 - $N$: the number of underlying data points.
 - $D$: the dimension of these points.
 - $K$: the dimension I want to the resulting data points to be in.

Heads up, in this post, I'm saying $X$ has shape $D \times N$.

```python
N = 1000
D = 20
K = 2

# rotate and uncenter the data
X = (2 * np.random.rand(D, D)) @ np.random.randn(D, N) + np.arange(D)[:, None]
```

## Centering matrix

In order to implement MDS, I'll need to use the *centering matrix*. This is given by the $n \times n$ matrix

$$C_n = \mathbb{I}_n - \frac 1 n \mathbb{1}_n \times \mathbb{1}_n^{\top}$$

where $\mathbb{I}_n$ is the identity matrix, and $\mathbb{1}_n \times \mathbb{1}_n^{\top}$ multiplies out to be an $n \times n$ matrix of 1's.
Given this, $X \, C_N$ removes the sample average by row and $C_D \, X$ removes the sample average by column.

To demonstrate this, I generated some 4-dimensional data with mean `[0, 1, 2, 3]`. Then I multiply it by the magical $\frac 1 n \mathbb{1}_n \times \mathbb{1}_n^{\top}$. Then I print a few columns of $X \, \frac 1 n \mathbb{1}_n \times \mathbb{1}_n^{\top}$ to show that it's roughly `[0, 1, 2, 3]`.

```python
# Generate data and then shift its center to [0, 1, 2, 3]
uncentered_data = np.random.randn(4, N) + np.arange(4)[:, None]

# build the magical 1/Ns matrix.
averaging_ones = (np.ones((N, N)))/N

# just show the first few rows
print("originally shifted by ",  np.arange(4))
print("averages found\n", (uncentered_data @ averaging_ones)[:, :6])
```

    originally shifted by  [0 1 2 3]
    averages found
     [[ 0.03647342  0.03647342  0.03647342  0.03647342  0.03647342  0.03647342]
     [ 1.01493437  1.01493437  1.01493437  1.01493437  1.01493437  1.01493437]
     [ 2.01166046  2.01166046  2.01166046  2.01166046  2.01166046  2.01166046]
     [ 2.97432857  2.97432857  2.97432857  2.97432857  2.97432857  2.97432857]]

    originally shifted by  [0 1 2 3]
    averages found
     [[-0.03951054 -0.03951054 -0.03951054 -0.03951054 -0.03951054 -0.03951054]
     [ 1.01209654  1.01209654  1.01209654  1.01209654  1.01209654  1.01209654]
     [ 2.01898754  2.01898754  2.01898754  2.01898754  2.01898754  2.01898754]
     [ 2.9924229   2.9924229   2.9924229   2.9924229   2.9924229   2.9924229 ]]

I'll make a function $C_n$ I can use later. I'll also create a centered set of $X$ to have handy.

```python
centering_matrix = lambda n: np.identity(n) -(np.ones((n, 1)) @ np.ones((1, n)))/n
centered_X = X @ centering_matrix(N)
```

## Detour: PCA on centered $X$

The goal of this post is to plot data points in two dimensions using only the distances between them. But since I have underlying high-dimensional data points $X$, as a warm-up I can apply PCA to them and plot the lower dimensional representation.

If singular value decomposition on the centered data points gives $X = USV^{\top}$, then the lower dimensional locations are given by $SV^{\top}$.

I show a little more about this in my [singular value decomposition demo](2018-05-16-SVD-demo.ipynb).

```python
# Do singular value decomposition
u, s, vh = np.linalg.svd(centered_X)
# Take the top K eigenvalues (np.linalg.svd orders eigenvalues)
pc_scores_from_X = np.diag(s[:K]) @ vh[:K]
```

```python
plt.plot(*pc_scores_from_X, '.')
plt.axis('equal')
maybe_save_plot('2018-05-20-pc-scores-from-x')
plt.show()
```

## Distance matrix

The distance matrix in this case is an $N \times N$ matrix where the element at $i, j$ is the Euclidean distance $\delta_{ij}$ between example $i$ and $j$, and $\delta_{ij} = ||x_i^2 - x_j^2||^2$.

I'll use `sklearn` for this. This gets approximately the same results as

```python
dist = np.zeros((N, N))
for i in range(N):
    for j in range(N):
        diff = X[:, i] - X[:, j]
        dist_squared = diff.T @ diff
        dist[i, j] = dist_squared
```

```python
distance_matrix = euclidean_distances(X.T)**2
```

For fun, I can plot a corner of the distance matrix. This shows what I'd expect: the diagonal elements are 0 because the distance between a point and itself is 0. It's otherwise symmetric and positive.

```python
fig, ax = plt.subplots()

ax.imshow(distance_matrix[:20, :20])
hide_ticks(ax)
maybe_save_plot('2018-05-20-distances')
plt.show()
```

The distances also don't care about whether the original data is centered or not. This makes sense since it's just the points relative to each other, not in relation to a coordinate system.

```python
distance_matrix_from_centered = euclidean_distances(centered_X.T)**2
assert np.allclose(distance_matrix, distance_matrix_from_centered)
```

## Gram matrix

This way of doing MDS works by getting the Gram matrix from the distance matrix. The Gram matrix is an $N \times N$ matrix given by $X^{\top}X$ using the centered $X$.

```python
actual_gram = centered_X.T @ centered_X
```

### Double centering the Gram matrix

One can convert a Gram matrix from uncentered data to the Gram matrix from centered data using **double centering**. This makes sense looking at the linear algebra.

$$C_n^{\top} (X^{\top} X) C_n = (X C_n)^{\top} (X C_n)$$

```python
uncentered_gram = X.T @ X

centered_gram = centering_matrix(N).T @ uncentered_gram @ centering_matrix(N)

assert np.allclose(actual_gram, centered_gram)
```

## PCA from the Gram matrix

It turns out once we have the Gram matrix, there's a way to compute the principal component scores from the Gram matrix.

You can see how this works by looking at the singular value decomposition.
When doing SVD on the observations $X$ directly, then $X$ becomes $X = USV^{\top}$ and the PC scores are given by $SV^\top$.

Here's what happens to $SV^{\top}$ in the Gram matrix.
Remember S is a diagonal, and $U$ and $V$ are orthonormal (so $U^{\top} = U^{-1}$ and $UU^{\top} = \mathbb{I}$):

\begin{align}
X^{\top}X &= G \\
 &= (USV^{\top})^{\top}(USV^{\top}) \\
 &= VSU^{\top}USV^{\top} \\
 &= VS^2V^{\top}.
\end{align}

Regular [eigendecomposition](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix) on the Gram matrix gives

$$G = Q\Lambda Q^{\top}.$$

Matching this up with $VS^2V^{\top}$ gives PC scores as $\sqrt{\Lambda} Q^{\top}$ using the top $K$ eigenvalues and eigenvectors.

```python
w, v = np.linalg.eig(centered_gram)
pc_score_from_gram_matrix = np.diag(np.sqrt(w[:K])) @ v.T[:K]
```

```python
plt.plot(*np.real(pc_score_from_gram_matrix), '.')
plt.axis('equal')
maybe_save_plot('2018-05-20-from-gram')
plt.show()
```

## Gram matrix from distance matrix.

There's one step missing: computing the Gram matrix from the distance matrix.

It turns out there's a clever way to get the centered Gram matrix out of the distances using the centering matrix again. It's given by

$$G = -\frac{1}{2} C_n \Delta C_n.$$

I can compare the Gram matrix given by this to the actual Gram matrix.

```python
gram_from_dist = -(centering_matrix(N) @ distance_matrix @ centering_matrix(N))/2

assert np.allclose(gram_from_dist, centered_gram)
```

## MDS

Now I can put it all together!

```python
def MDS(distance_matrix, K):
    # Double checking that the matrix is the right size
    N = distance_matrix.shape[0]
    assert distance_matrix.shape[0] == distance_matrix.shape[1], 'dist should be a square matrix, but it\'s {}x{}'.format(dist.shape)

    # Compute the Gram matrix from the distance_matrix
    gram_from_dist = -(centering_matrix(N) @ distance_matrix @ centering_matrix(N))/2

    # Compute the PC scores from Gram matrix
    w, v = np.linalg.eig(gram_from_dist)
    # Double check the eigenvalues are positive. While they will be for
    # actual distance matrices, this becomes a problem if we extend this
    # to other types of symmetric matrices that may not be positive semi-definite.
    assert np.all(w[:K] > 0)
    proj = np.diag(np.sqrt(w[:K])) @ v.T[:K]

    return proj

pc_scores_from_distance_matrix = MDS(distance_matrix, K)
```

## Viewing them all

When I plot these, I expect the points to be in the same place, though they might be rotated.

```python
fig, axs = plt.subplots(1, 3, figsize=(12, 4), sharex=True, sharey=True)

axs[0].plot(*pc_scores_from_X, '.')
axs[0].set_title("Centered X")
axs[0].axis('equal')

axs[1].plot(*np.real(pc_score_from_gram_matrix), '.')
axs[1].set_title("Gram matrix")

axs[2].plot(*np.real(pc_scores_from_distance_matrix), '.')
axs[2].set_title("Distance matrix")

maybe_save_plot('2018-05-20-comparison')
plt.show()
```

## North American Cities example

Inspired by [this video](https://www.youtube.com/watch?v=qZ_QHXJ29iw) let's try MDS using the distances between cities.

[This website](https://people.sc.fsu.edu/~jburkardt/datasets/cities/cities.html) has data in this form. (Heads up, it's licensed under [GNU LGPL](https://people.sc.fsu.edu/~jburkardt/txt/gnu_lgpl.txt))
I'll use one of the datasets that measure distances using latitude/longitude, rather than road distance. USCA312 is a dataset of 312 cities in the US and Canada. This MDS approach assumes Euclidean distances, but distances between locations on Earth are non-Euclidean, so there will be some approximation.

I'll first download the data and process it.

```python
dataset_name = "usca312"

import requests

def get_data_from_fsu(url):
    response = requests.get(url)
    response_lines = response.text.split('\n')
    for i, line in enumerate(response_lines):
        if not line.startswith('#'):
            break
    return response_lines[i:]

url = "https://people.sc.fsu.edu/~jburkardt/datasets/cities/"

raw_city_labels_data = get_data_from_fsu("{}{}_name.txt".format(url, dataset_name))
raw_city_dists_data = get_data_from_fsu("{}{}_dist.txt".format(url, dataset_name))
```

```python
# the last element is a blank character
assert raw_city_labels_data[-1] == ''
city_labels = raw_city_labels_data[:-1]

# The txt file splits long rows up, and it's a tiny bit easier to
# just make a big long list and reshape it
raw_city_dists_list = sum([
    list(map(int, line.split()))
    for line in raw_city_dists_data
], [])

city_dist_matrix = np.array(raw_city_dists_list).reshape(
    len(city_labels),
    len(city_labels)
)
```

### Computing PC Scores

Now that I have the distance matrix loaded, I'll run the `MDS` function defined above!

```python
locations = MDS(city_dist_matrix, K)
```

Now I'll plot and label the states/provinces. I strip the city names out of the labels to make it a little easier to read.

```python
plt.clf()
fig, ax = plt.subplots(1, 1, figsize=(16, 12))

plt.plot(locations[0], locations[1], 'ok', alpha=.2)

for label, xy in zip(city_labels, locations.T):
    plt.annotate(label.split(',')[1].strip(), xy, fontsize=14)

hide_ticks(ax)
plt.axis('equal')

maybe_save_plot('2018-05-20-north-america')
plt.show()
```

One more plotting alternative is to plot one label per state/province.

```python
state_or_province_labels = {}
for i, label in enumerate(city_labels):
    state_or_province = label.split(',')[1].strip()
    # it's fine if we overwrite states
    state_or_province_labels[state_or_province] = i

plt.clf()
fig, ax = plt.subplots(1, 1, figsize=(16, 12))

plt.plot(locations[0], locations[1], 'ok', alpha=.2)

annotation_subset = list(state_or_province_labels.values())
for label, idx in state_or_province_labels.items():
    plt.annotate(label, locations[:, idx], fontsize=16, weight='bold')

hide_ticks(ax)
plt.axis('equal')

maybe_save_plot('2018-05-20-north-america-per-state-province')
plt.show()
```

## See Also

 - MDS reminds me of [t-sne](https://distill.pub/2016/misread-tsne/), though MDS works on distance matrices.
 - [DME](http://www.inf.ed.ac.uk/teaching/courses/dme/) is the the course that taught me this.
 - [MDS on Wikipedia](https://en.wikipedia.org/wiki/Multidimensional_scaling).

```python

```